Test results upload from Azure

Test results upload from Azure

Azure’s upload workflow uses allurectl.

Foreword

This very article describes the basics – how to create a pipeline in Azure DevOps and describes in details on how to upload test results from Azure DevOps pipelines.

If you need to get more information on Azure pipelines, you need to refer to dedicated resources like official documentation pages.

Creation of a pipeline

  1. In your project go to pipelines.

Azure go to pipelines

  1. Then click special button to create a new pipeline.

Azure create new pipeline

  1. Select the source for your code

    1. Select the VCS from the list
    2. Select the repository in VCS
  2. You either need to use an existing yaml file with pipeline instructions or create a new one

Azure create new pipeline

Authenticate Azure’s build job

For allurectl to send the test results to Allure TestOps server you need it to be authenticated.

To authenticate allurectl you need to perform 2 actions:

  1. Create API token on Allure TestOps side
  2. Provide API token to allurectl alongside with the information on Endpoint and used project where you will send the test results to.

API token

For a user on Allure TestOps side you need to create API token.

This API token will be used by allurectl running in Azure pipeline to authenticate on Allure TestOps server.

  1. Log in to Allure TestOps with the account you are going to use to upload test results.
  2. Go to user’s profile.

Allure go to profile

  1. In the section API tokens click a green button that says Create.
  2. Name your API token and click Submit. Always give meaningful names to all the configuration parameters.

Allure go to profile

Allure will randomly generate the token and show you in modal window.

  1. Copy the token and save it in a safe place as it cannot be retrieved by Allure TestOps means.

Environment variables

allurectl uses environment variables to get the information on Allure TestOps endpoint, project and API token.

To add the required variables you need to click Variables button when configuring your pipeline.

Azure add variables to the pipeline

The following variables need to be added:

  • ALLURE_ENDPOINT - URL of Allure TestOps server.
  • ALLURE_PROJECT_ID - the ID of the Allure TestOps project.
  • ALLURE_TOKEN - the authentication token you created on the previous steps.

For the secrets you need to use “Keep this value secret” option.

Azure add variables to the pipeline

If you need to be able to update the value of the variable during the pipeline run, you need to use “Let users override this value when running this pipeline” option.

This also means, if you are using some environment variables in your tests like browser name, URL of a host to run the tests on etc., and you need to pass this information from Allure TestOps, you need to add these variables to the pipeline variables as well (see the example below).

Finally, you’ll have the set of variables needed for allurectl to send the data to Allure TestOps.

Azure add variables to the pipeline

There are other variables you can and need to set, but these three are absolutely essential and it’s better to keep these at pipeline level as we just showed.

That’s it. Now, you can configure your build jobs to send test results to Allure TestOps server.

Pipelines

This section describes mandatory information and commands to be included into your pipeline for test results upload.

Pipeline example

In your Azure pipeline, you need to prepare all the data and tools for allurectl to execute the tests and upload the test results to Allure TestOps server.

Here, is an example of pipeline (see the comments below the example):

<snip>
steps:
- task: [email protected]3
  displayName: 'Install pytest, install allure-pytest'
  inputs:
    targetType: 'inline'
    script: |
      pip install pytest allure-pytest

- task: [email protected]3
  displayName: 'Download and prepare allurectl'
  inputs:
    targetType: 'inline'
    script: |
      wget https://github.com/allure-framework/allurectl/releases/latest/download/allurectl_linux_386 -O ./allurectl
      chmod +x ./allurectl

- script: |
    ./allurectl job-run plan --output-file $(ALLURE_TESTPLAN_PATH)
    ./allurectl watch -- pytest --alluredir=$(ALLURE_RESULTS)

  displayName: 'running pytest tests'
  env:
    ALLURE_ENDPOINT: $(ALLURE_ENDPOINT)
    ALLURE_PROJECT_ID: $(ALLURE_PROJECT_ID)
    ALLURE_TOKEN: $(ALLURE_TOKEN)
    ALLURE_TESTPLAN_PATH: $(ALLURE_TESTPLAN_PATH)
    ALLURE_RESULTS: $(ALLURE_RESULTS)
    ALLURE_LAUNCH_NAME: "$(Build.DefinitionName)-$(Build.BuildNumber)"
    ALLURE_LAUNCH_TAGS: "$(Build.SourceBranchName), azure"
    TEST_BRANCH: $(Build.SourceBranchName)
    AZURE_BROWSER: $(AZURE_BROWSER)
    AZURE_HOST: $(AZURE_HOST)

So, what you need to do in your pipeline:

  1. Prepare the base for your tests, i.e. install all the tools and dependencies (first task: [email protected]).
  2. Download appropriate allurectl binary to your pipeline. See here the available ones. , (second task: [email protected]).
  3. Make allurectl executable (second task: [email protected]).
  4. Execute allurectl to trigger your tests and upload the test results.
    1. The construct ./allurectl job-run plan --output-file $(ALLURE_TESTPLAN_PATH) is required to enable selective tests run on pipeline level (also depends on your test framework integration with Allure Framework). We advise defining the env variable $(ALLURE_TESTPLAN_PATH) on pipeline’s level.
    2. The command watch is the preferred way to upload the test results to Allure TestOps.
    3. Most convenient way to provide all the data for proper upload is to use environment variables, the upload command will have clearer format and it’ll be easier to debug the pipeline.
  5. Provide environment variables for the script with the test execution and test results upload.
    1. ALLURE_ENDPOINT, ALLURE_PROJECT_ID, ALLURE_TOKEN, ALLURE_RESULTS are generally required to upload the test results data. These are processed by allurectl automatically.
    2. ALLURE_TESTPLAN_PATH is required for selective test run to work (also depends on Allure Framework adaptor capabilities). Processed by allurectl automatically.
    3. ALLURE_LAUNCH_NAME defines how launches will be displayed in Allure TestOps UI.
    4. ALLURE_LAUNCH_TAGS provides additional metadata on a launch and also used in Allure TestOps UI as well as for searching and filtering of the launches.
    5. TEST_BRANCH is environment variable used to collect the data on VCS branch name used to run the tests in the pipeline. This information is eventually used for triggering jobs from Allure TestOps side.
    6. AZURE_BROWSER, AZURE_HOST are env variables used to pass additional data to your tests. This information is also used to provide meta-data on launches and will be displayed in the UI.

The usage of the environment variables is described here.