How can we consolidate results from different CI pipelines to a single launch

Task

We’re working on a software product composed of multiple modules, all bundled into a single release version. The testing setup is diverse — some test suites are automated using GitLab, others via Jenkins.

Since different modules (within the same release) may be tested through different CI pipelines, we're looking for a way to consolidate all results into a single launch per release.

How to

If your pipelines can be triggered independently, it's better to trigger these from Allure TestOps and avoid using this very guide.

You can create a test plan with tests from different pipelines and CIs, and then run such test plan. Allure TestOps will request pipelines execution and all these will report data to a single launch without any additional actions and pipelines modifications

Concepts and entities in Allure TestOps

Launch — is a container for test results (for automated tests and manual tests together).

Job Run — is a container for automated (only) test results executed in the scope of a pipeline execution/run inside a Launch. Each job run has a unique ID.

Job Run is always associated with a Launch. If you have Job Run URL, then you will be redirected to the Launch, that contains this Job Run. If several launches are merged into a single launch, then Job Run URL will still point to the correct resulting Launch, meanwhile the URL of the initial launch where Job Run was created, will become invalid (won't exist anymore) except for the cases when the said launch is the resulting one.

Job – is a connection between Job Run and CI/CD pipeline configuration.

One Launch can contain many Job Runs associated with different pipelines of the same CI/CD or pipelines from different CI/CD systems.

Manual tests results always are collected under Launch and never collected under a Job Run.

Automated tests results executed in the scope of a pipeline execution of CI/CD that is properly integrated with Allure TestOps are always collected under a Job Run.

If automated tests are executed from Allure TestOps side, then Job Run is first created on Allure TestOps side, the information about the test cases ID that need to be executed on CI side are linked to the Job Run.

If automated tests are executed from are executed from CI/CD side, then either allurectl or a plugin for CI (e.g. Allure TestOps plug-in for Jenkins) creates i) a new launch ii) Job Rin inside the launch. Both entities have their unique IDs.

Case #1. Pipelines are triggered by CI logic

Getting Launch information and job run information

Generally, the process of collecting the test results must look like follows:

1. There is an initial job in a pipeline; it creates a new launch and a job run.
   • The commands that can create a new launch are:
     • allurectl watch
     • allurectl upload
     • allurectl job-run start
2. Retrieve the information about Launch and Job Run.
   • information can be retrieved by executing of the command export $(allurectl job run env)
   • command will create environment variables ALLURE_LAUNCH_ID and ALLURE_JOB_RUN_ID
   • you can get the values and variables using standard *ux commands like env or printenv
3. The values of the both variables need to be passed to the consecutive jobs or pipelines or other CI pipelines.
   • in other words before the execution of tests will start in consecutive pipelines you need to create environment variables ALLURE_LAUNCH_ID and ALLURE_JOB_RUN_ID in these pipelines and set their values to the ones received in the step #2.

After you have created ALLURE_LAUNCH_ID and ALLURE_JOB_RUN_ID you can stat allurectl.

First allurectl gets the pipeline context and looks for the variables required for the upload.

If there is anything with ALLURE_, allurectl will try using the values from these variables and will upload the test results to created launch and Job Run.

Case #2. Pipelines are triggered by Allure TestOps

Prerequisites

1. Your tests must be executed at least once from the pipeline side and such launch must be closed.
2. Integration with a CI must be configured for bi-directional work.
   1. Integration with CI must be added.
   2. Job for the pipeline must be configured and integration from 2.1 must be added to the Job as a build server.
   3. Job must be marked as "Job can be used to run tests"

Test cases selection or test plan

There are two ways you can create a launch, that sill collect the data from different CI/pipelines.

1. In the test cases list select the tests you need, then use the action to run the tests.
2. Create a test plan, that would contain all the tests.

Each automated test case is automatically linked to the most recent job was used to upload the test results. This information will be used for the request of the execution.

In each test plan, there is Executors section (see below), that contains the information about Job (i.e. which pipeline) will be used for the execution of the test cases.

Execution

When either selected list of test cases or a test plan is executed, you will see the Launch creation form. You can provide additional data to the execution such as tags, issue links, simple links to mark this launch with something that could point to a release or any other activity you would like to easily find.

Allure TestOps will create a launch and Job Runs inside this launch. Each Job (CI pipeline) then will report the test results to the same launch without any alteration of the pipelines.