Test results upload from Azure

This integration is a specific case of Integrations with external systems.

Allure Testops integration with Azure is bidirectional, and Azure can be integrated as a CI system, and as an issue tracker, this means you can do the following:

• CI: Upload test results from a Azure pipelines (Azure > Allure Testops).
• CI: Trigger a Azure pipeline (Allure Testops > Azure).
• Issue tacker: link test cases to Azure issues from tests code (via labels).
• Issue tacker: link Launches to Azure issues (broken in 4.16.x - 4.17.x).
• Issue tacker: create bug reports in Azure from Allure Testops defects.

Azure's upload workflow uses allurectl.

Foreword

This very article describes the basics – how to use Azure DevOps pipeline to upload the test results from the pipeline to ATO Allure Testops and trigger the pipelines from Allure Testops.

If you need to get more information on Azure pipelines, you need to refer to dedicated resources like official documentation pages.

Creation of a pipeline

1. In your project go to pipelines.

2. Then click special button to create a new pipeline.

3. Select the source for your code

   • Select the VCS from the list
   • Select the repository in VCS

4. You either need to use an existing yaml file with pipeline instructions or create a new one

Authenticate Azure's build job

For allurectl to send the test results to Allure Testops server you need it to be authenticated.

To authenticate allurectl you need to perform 2 actions:

1. Create API token on Allure Testops side
2. Provide API token to allurectl alongside with the information on Endpoint and used project where you will send the test results to.

API token

For a user on Allure Testops side you need to create API token.

This API token will be used by allurectl running in Azure pipeline to authenticate on Allure Testops server.

1. Log in to Allure Testops with the account you are going to use to upload test results.
2. Go to user's profile.

3. In the section API tokens click a green button that says Create.
4. Name your API token and click Submit. Always give meaningful names to all the configuration parameters.

Allure will randomly generate the token and show you in modal window.

5. Copy the token and save it in a safe place as it cannot be retrieved by Allure Testops means.

Environment variables

allurectl uses environment variables to get the information on Allure Testops endpoint, project and API token.

The variables containing data for the test results upload are better to be placed in Variables area of Azure pipeline, this will allow masking secrets like API tokens.

To add the variables you need to click Variables button when configuring your pipeline.

All of the following variables must be added to allow the test results upload to Allure Testops:

• ALLURE_ENDPOINT - URL of Allure Testops server.
• ALLURE_PROJECT_ID - the ID of the Allure Testops project where you want to send the test results to.
• ALLURE_TOKEN - the API token you created on the previous steps, it's used to authenticate pipeline to add data to Allure Testops
• ALLURE_JOB_RUN_ID - this variable is an internal variable used by allurectl for the proper mapping of the pipeline entities to Allure Testops entities.

For the secrets (e.g. API token) you need to use "Keep this value secret" option.

ALLURE_JOB_RUN_ID variable

This is a special service variable used by allurectl and Allure Testops.

ALLURE_JOB_RUN_ID needs to be set as follows:

• The value of ALLURE_JOB_RUN_ID needs to be empty
• "Let users override this value when running this pipeline" needs to be checked

Other variables

Please refer to the description of allurectl to add other variables you might need in the scope of your test result upload.

If you need to be able to update the value of the variable during the pipeline run, you need to use "Let users override this value when running this pipeline" option.

This also means, if you are using some environment variables in your tests like browser name, URL of a host to run the tests on etc., and you need to pass this information from Allure Testops, you need to add these variables to the pipeline variables as well (see the example below).

That's it. Now, you can configure your build jobs to send test results to Allure Testops server.

Pipelines

This section describes mandatory information and commands to be included into your pipeline for test results upload.

Pipeline example

In your Azure pipeline, you need to prepare all the data and tools for allurectl to execute the tests and upload the test results to Allure Testops server.

Here, is an example of pipeline (see the comments below the example):

<snip>
steps:
- task: Bash@3
  displayName: 'Install pytest, install allure-pytest'
  inputs:
    targetType: 'inline'
    script: |
      pip install pytest allure-pytest

- task: Bash@3
  displayName: 'Download and prepare allurectl'
  inputs:
    targetType: 'inline'
    script: |
      wget https://github.com/allure-framework/allurectl/releases/latest/download/allurectl_linux_386 -O ./allurectl
      chmod +x ./allurectl

- script: |
    ./allurectl watch -- pytest --alluredir=$(ALLURE_RESULTS)

  displayName: 'running pytest tests'
  env:
    ALLURE_ENDPOINT: $(ALLURE_ENDPOINT)
    ALLURE_PROJECT_ID: $(ALLURE_PROJECT_ID)
    ALLURE_TOKEN: $(ALLURE_TOKEN)
    ALLURE_RESULTS: $(ALLURE_RESULTS)
    ALLURE_LAUNCH_NAME: "$(Build.DefinitionName)-$(Build.BuildNumber)"
    ALLURE_LAUNCH_TAGS: "$(Build.SourceBranchName), azure"
    TEST_BRANCH: $(Build.SourceBranchName)
    AZURE_BROWSER: $(AZURE_BROWSER)
    AZURE_HOST: $(AZURE_HOST)

So, what you need to do in your pipeline:

1. Prepare the base for your tests, i.e. install all the tools and dependencies (first task: Bash@3).
2. Download appropriate allurectl binary to your pipeline. See here the available ones., (second task: Bash@3).
3. Make allurectl executable (second task: Bash@3).
4. Execute allurectl to trigger your tests and upload the test results.
   1. The command watch is the preferred way to upload the test results to Allure Testops.
   2. Most convenient way to provide all the data for proper upload is to use environment variables, the upload command will have clearer format and it'll be easier to debug the pipeline.
5. Provide environment variables for the script with the test execution and test results upload.
   1. ALLURE_ENDPOINT, ALLURE_PROJECT_ID, ALLURE_TOKEN, ALLURE_RESULTS are generally required to upload the test results data. These are processed by allurectl automatically.
   2. ALLURE_LAUNCH_NAME defines how launches will be displayed in Allure Testops UI.
   3. ALLURE_LAUNCH_TAGS provides additional metadata on a launch and also used in Allure Testops UI as well as for searching and filtering of the launches.
   4. TEST_BRANCH is environment variable used to collect the data on VCS branch name used to run the tests in the pipeline. This information is eventually used for triggering jobs from Allure Testops side.
   5. AZURE_BROWSER, AZURE_HOST are env variables used to pass additional data to your tests. This information is also used to provide meta-data on launches and will be displayed in the UI.

The usage of the environment variables is described here.

Gathering environment information from Azure pipelines

When we say environment we consider the context of a pipeline or context of a job inside a pipeline.

Context is usually stored in the environment variables (predefined Azure pipeline variables or user-defined ones) which can be used in Allure Testops to distinguish results received from different jobs or for different test environments (here we use the information about a application under test, browser used and so on, i.e. data you pass to your tests).

Environment information can be used to

• filter the test results generated with this environment information, e.g. you can filter out the test results executed using a certain browser name or executed in certain job of a pipeline
• use the collected information to trigger CI pipelines with certain environment parameters

Preconditions

Before using, the environment variables need to be created in the Environment section on the Global level of Allure Testops, then they can be used in projects to map pipelines' env data to them.

To properly collect environment variables from Azure pipelines, first you must understand what environment data you have in your pipeline.

1. Go to pipeline Editor in an Azure pipeline (to edit azure-pipelines.yml file).
2. after allurectl execution add printenv command.

The command printenv can be deleted later on or kept for troubleshooting purposes.

Now, when you start the pipeline, the pipeline execution console will display all environment variables, so you can create mapping in TestOps.

For example we will add the mapping for Branch variable created on the global level (by an admin in Administration section).

The variable containing the branch name is created by us and its value is being taken from the predefined variables.

The predefined variables reference can be found here.

On Allure Testops project side

1. Go to Project > Settings > Environment and click Create.

1. Add variable name from Azure pipeline to Mapping Key field (TEST_BRANCH).
2. In the drop-down list select the Environment variable that was created on the Global level for mapping (Branch).
3. Click Submit.

Now, if you start the pipeline, you will see the value of the added variable on the page with Launches in the Environments section.

Now, we can see and use the branch name in filters and AQL when creating the dashboards.

All other Environment variables are processed in the same way:

1. Global environment must be added
2. Mapping of Env var name as it is used in a pipeline to a global Env variable must be added on the project level.

The processing of the environment starts after the mapping is created and does not include the launches created before the mapping creation.

Triggering Azure pipelines from Allure Testops

Job and its configuration

Job is an Allure TestOps entity linking your Allure TestOps project with the pipeline where you send the test results from.

Job is created by Allure TestOps automatically after the first test results upload from a pipeline. Even automatically created job requires configuration.

The configuration of the job for the triggering of the Azure workflows is only possible when the integration with Azure has been completed.

Configuring the integration with Azure Devops

Integrations recap

As any other integration Wrike integration first needs to be enabled globally in Allure Testops instance, and then enabled integration needs to be added to the project where you are going to use this integration. Whole process is described in the integrations overview.

Enable Azure globally

You need to be Allure Testops administrator.

1. Go to Administration > Integrations
2. Click the button Add integration
3. Find Azure banner in the list and click it.
4. Give the integration clear distinguishable name.
5. Add the endpoint https://dev.azure.com/organization_name
   • the endpoint must contain https://dev.azure.com
   • the endpoint must contain the organization name
6. Save changes.

Create Azure token

1. Go to the address https://dev.azure.com/{organization_name}/_usersSettings/tokens
   • {organization_name} is the organization name of your account in Azure, it's different for each organization.
2. In the Personal Access Tokens section click New Token
3. Fill the token's parameters
   1. Name
   2. Organization (or leave as it is)
   3. Expiration date
   4. Define the scopes
      1. Build: Read and Execute (for working with the pipelines).
      2. Work Items: Read, Write and Manage (for working with the task tracker).
   5. Save the token details.
   6. Copy the token to proceed to next steps.

Add Azure integration to your project

You need to be a project Owner or an Administrator

1. Go to the project settings
2. Go to Integrations section
3. Locate Azure integration under Available integrations
4. Click Add integration
5. Add Azure's personal access token string to Azure token text field
6. Click Test connection
   1. You must see "Connection established" notification
   2. If not, then check your token is correct.
7. Save the integration.

Now we can finalise the configuration of the Job created after the test results upload.

Finalising the Job configuration

Prerequisites

1. Job needs to be created (run tests once from CI side).
2. Integration with Azure needs to be added to your project.

Configuring the job

1. Go to Jobs section of your project in Allure Testops.
2. Select the Job you want to use for triggering the Azure pipelines.
3. Click Job actions menu button (three vertical dots) and click Configure.

The job configuration modal will look like follows.

To complete the Job's configuration you need to fill the sections

1. Build server – this is the name of the integration with Azure as you named it in the global Integrations.
2. Job must be allowed to run tests, i.e. this allows Job to trigger pipelines.
3. You need to add the Job's parameters.

Job parameters

Job parameter is an environment variable you want to control from Allure Testops side, e.g. if you want to control the environment your tests are running on, then the variables need to be added to Job parameters.

Examples of such variables are

• Branch name,
• browser to be used,
• size of the screen,
• test bed reference (prod, preprod, qa, etc.)

When configuring the Job parameters we go from left to right and add the following information

1. Name of the Environment variable as it appears in an Azure pipeline.
   • Let users override this value when running this pipeline needs to be checked for such variable
2. Default value. This value will be used if you won't alter its value when triggering a pipeline.
3. Name of Global environment variable configured in Allure Testops. This will be used as the source for the values suggestions.

Now, trigger your pipeline

From Test Cases list

1. In the test cases section select the test cases you want to execute
2. Select the bulk operations for the test cases (becomes active on selection of test cases)
3. Select Run action

From Test plans

1. In the Test Plans section locate a test plan
2. Click on Run action of the desired test plan

Trigger a Job

1. In Jobs section locate a Job
2. Click Run job button

Pass the non-default environment data

Pipeline execution corresponds to a single launch in Allure Testops

When you trigger a Job using any way described above (test cases, test plans, jobs), you will see the launch creation modal dialogue.

When creating a launch you need to fill the following fields

1. Name - add a name or skip, Allure Testops will use the placeholder.
2. Tags - add for better search and grouping (e.g. in Dashboards) experience.
3. Issues - add if launch is a milestone and needs to be linked with particular task in an issue tracker, or skip.
4. Links - provide links related to the launch being created. This also can be skipped.
5. Environment - add alternative environment for tests execution.
6. Tests - select tests to be executed (this will be visible if executing from Jobs)

Alternative environment

The set of the environment variables is created by clicking Add button. UI will create a section where you need to select global variables used in the Job's parameters section and the values you want to pass to pipeline we are going to create.

You can add several sets of environments and Allure Testops will request several pipelines with several sets of non-default environments to be executed on CI side.