Test results upload from Azure

Azure's upload workflow uses allurectl.

Foreword

This very article describes the basics – how to create a pipeline in Azure DevOps and describes in details on how to upload test results from Azure DevOps pipelines.

If you need to get more information on Azure pipelines, you need to refer to dedicated resources like official documentation pages.

Creation of a pipeline

1. In your project go to pipelines.

2. Then click special button to create a new pipeline.

3. Select the source for your code

   1. Select the VCS from the list
   2. Select the repository in VCS

4. You either need to use an existing yaml file with pipeline instructions or create a new one

Authenticate Azure's build job

For allurectl to send the test results to Allure TestOps server you need it to be authenticated.

To authenticate allurectl you need to perform 2 actions:

1. Create API token on Allure TestOps side
2. Provide API token to allurectl alongside with the information on Endpoint and used project where you will send the test results to.

API token

For a user on Allure TestOps side you need to create API token.

This API token will be used by allurectl running in Azure pipeline to authenticate on Allure TestOps server.

1. Log in to Allure TestOps with the account you are going to use to upload test results.
2. Go to user's profile.

3. In the section API tokens click a green button that says Create.
4. Name your API token and click Submit. Always give meaningful names to all the configuration parameters.

Allure will randomly generate the token and show you in modal window.

5. Copy the token and save it in a safe place as it cannot be retrieved by Allure TestOps means.

Environment variables

allurectl uses environment variables to get the information on Allure TestOps endpoint, project and API token.

To add the required variables you need to click Variables button when configuring your pipeline.

The following variables need to be added to allow the test results upload to ATO Allure Test:

• ALLURE_ENDPOINT - URL of Allure TestOps server.
• ALLURE_PROJECT_ID - the ID of the Allure TestOps project.
• ALLURE_TOKEN - the authentication token you created on the previous steps.
• ALLURE_JOB_RUN_ID - this variable is a service one, it'll be used by allurectl for the proper mapping of the pipeline entities to Allure TestOps entities.

For the secrets you need to use "Keep this value secret" option.

ALLURE_JOB_RUN_ID service variable

This is a special service variable used by allurectl.

ALLURE_JOB_RUN_ID needs to be set as follows:

• The value of ALLURE_JOB_RUN_ID needs to be empty
• "Let users override this value when running this pipeline" needs to be checked

Other service variables

Please refer to the description of allurectl to add other variables you might need.

If you need to be able to update the value of the variable during the pipeline run, you need to use "Let users override this value when running this pipeline" option.

This also means, if you are using some environment variables in your tests like browser name, URL of a host to run the tests on etc., and you need to pass this information from Allure TestOps, you need to add these variables to the pipeline variables as well (see the example below).

That's it. Now, you can configure your build jobs to send test results to Allure TestOps server.

Pipelines

This section describes mandatory information and commands to be included into your pipeline for test results upload.

Pipeline example

In your Azure pipeline, you need to prepare all the data and tools for allurectl to execute the tests and upload the test results to Allure TestOps server.

Here, is an example of pipeline (see the comments below the example):

<snip>
steps:
- task: [email protected]
  displayName: 'Install pytest, install allure-pytest'
  inputs:
    targetType: 'inline'
    script: |
      pip install pytest allure-pytest

- task: [email protected]
  displayName: 'Download and prepare allurectl'
  inputs:
    targetType: 'inline'
    script: |
      wget https://github.com/allure-framework/allurectl/releases/latest/download/allurectl_linux_386 -O ./allurectl
      chmod +x ./allurectl

- script: |
    ./allurectl job-run plan --output-file $(ALLURE_TESTPLAN_PATH)
    ./allurectl watch -- pytest --alluredir=$(ALLURE_RESULTS)

  displayName: 'running pytest tests'
  env:
    ALLURE_ENDPOINT: $(ALLURE_ENDPOINT)
    ALLURE_PROJECT_ID: $(ALLURE_PROJECT_ID)
    ALLURE_TOKEN: $(ALLURE_TOKEN)
    ALLURE_TESTPLAN_PATH: $(ALLURE_TESTPLAN_PATH)
    ALLURE_RESULTS: $(ALLURE_RESULTS)
    ALLURE_LAUNCH_NAME: "$(Build.DefinitionName)-$(Build.BuildNumber)"
    ALLURE_LAUNCH_TAGS: "$(Build.SourceBranchName), azure"
    TEST_BRANCH: $(Build.SourceBranchName)
    AZURE_BROWSER: $(AZURE_BROWSER)
    AZURE_HOST: $(AZURE_HOST)

So, what you need to do in your pipeline:

1. Prepare the base for your tests, i.e. install all the tools and dependencies (first task: [email protected]).
2. Download appropriate allurectl binary to your pipeline. See here the available ones., (second task: [email protected]).
3. Make allurectl executable (second task: [email protected]).
4. Execute allurectl to trigger your tests and upload the test results.
   1. The construct ./allurectl job-run plan --output-file $(ALLURE_TESTPLAN_PATH) is required to enable selective tests run on pipeline level (also depends on your test framework integration with Allure Framework). We advise defining the env variable $(ALLURE_TESTPLAN_PATH) on pipeline's level.
   2. The command watch is the preferred way to upload the test results to Allure TestOps.
   3. Most convenient way to provide all the data for proper upload is to use environment variables, the upload command will have clearer format and it'll be easier to debug the pipeline.
5. Provide environment variables for the script with the test execution and test results upload.
   1. ALLURE_ENDPOINT, ALLURE_PROJECT_ID, ALLURE_TOKEN, ALLURE_RESULTS are generally required to upload the test results data. These are processed by allurectl automatically.
   2. ALLURE_TESTPLAN_PATH is required for selective test run to work (also depends on Allure Framework adaptor capabilities). Processed by allurectl automatically.
   3. ALLURE_LAUNCH_NAME defines how launches will be displayed in Allure TestOps UI.
   4. ALLURE_LAUNCH_TAGS provides additional metadata on a launch and also used in Allure TestOps UI as well as for searching and filtering of the launches.
   5. TEST_BRANCH is environment variable used to collect the data on VCS branch name used to run the tests in the pipeline. This information is eventually used for triggering jobs from Allure TestOps side.
   6. AZURE_BROWSER, AZURE_HOST are env variables used to pass additional data to your tests. This information is also used to provide meta-data on launches and will be displayed in the UI.

The usage of the environment variables is described here.