TestOps logo
Try TestOps

Integration with Azure DevOps

This page describes how to set up both Allure TestOps and Azure DevOps in a project that uses Azure Pipelines to run tests.

Like all other CI integrations for Allure TestOps, this involves communication in two directions.

  • A job in an Azure pipeline will use allurectl to send its status and test results to Allure TestOps.
  • Allure TestOps will use Azure DevOps API to trigger new pipelines, optionally passing a test plan.

Once a pipeline run stops, the corresponding launch in Allure TestOps may be closed either manually or automatically, according to the Auto close policy.

To enable Allure TestOps support in your Azure DevOps pipeline:

  1. enable sending data from Azure DevOps,
  2. enable triggering Azure pipelines,
  3. parametrize jobs (if necessary).

1. Enable sending data from Azure DevOps

For Azure DevOps to send run statuses and test results to Allure TestOps, you need to generate an authentication token in Allure TestOps, add the token to Azure DevOps, and modify the pipeline itself.

Then, run and check the pipeline to make sure that everything works so far.

1.1. Create a token in Allure TestOps

  1. In Allure TestOps, click on your avatar and go to Your profile.

  2. Under the API tokens section, click Create.

  3. Enter a Token name (e.g., “Token for Azure DevOps”), then click Submit.

  4. In the dialog that appears, click the Copy icon to copy the token into clipboard.

    You will need this token on the next step.

1.2. Specify the token in Azure DevOps

  1. In Azure DevOps, go to Pipelines and select the pipeline you need.

  2. In the top right corner, click Edit.

  3. In the top right, click the Variables button.

  4. Click New variable or +. In the dialog that appears, fill in the fields:

    • Name — “ALLURE_TOKEN”.
    • Value — the API token that you got on step 1.1.
    • Keep this value secret — must be checked.
    • Let users override this value when running this pipeline — must not be checked.

    Click OK.

  5. Click + again. In the dialog that appears, fill in the fields:

    • Name — “ALLURE_JOB_RUN_ID”.
    • Value — must be blank.
    • Keep this value secret — must not be checked.
    • Let users override this value when running this pipeline — must be checked.

    Click OK.

  6. Click Save to apply the changes and close the dialog.

1.3. Modify the pipeline

Make the following changes in the azure-pipelines.yml file in your repository. You can do it either in a local directory or in the Azure's web-based text editor.

  1. At the beginning of the steps section, add a new step that downloads the allurectl tool and makes it executable.

  2. In the step that runs tests, add or extend the env block. It must include the following variables:

    • ALLURE_ENDPOINT — the URL of the Allure TestOps server.
    • ALLURE_PROJECT_ID — the ID of the Allure TestOps project.
    • ALLURE_RESULTS — the path to the test results directory, e.g., “build/allure-results”. If there are multiple test result directories in your project, you can separate them with commas or use a wildcard pattern, e.g., “modules/*/build/allure-results”.
    • ALLURE_TOKEN — $(ALLURE_TOKEN).

  3. Wrap the command that runs the tests into the ./allurectl watch command.

Example

Assume we have a Java project with a azure-pipelines.yml file like this:

steps:
  - script: ./gradlew clean test
    displayName: Run tests

With Allure TestOps integration, the file will look something like this:

steps:
  - bash: curl -fsSL https://github.com/allure-framework/allurectl/releases/latest/download/allurectl_linux_amd64 -o allurectl && chmod +x allurectl
    displayName: Download allurectl

  - script: ./allurectl watch -- ./gradlew clean test
    displayName: Run tests
    env:
      ALLURE_ENDPOINT: https://demo.testops.cloud
      ALLURE_PROJECT_ID: 1
      ALLURE_RESULTS: build/allure-results
      ALLURE_TOKEN: $(ALLURE_TOKEN)

1.4. Run and check the pipeline

  1. In Azure DevOps, open the project and go to the pipeline run triggered by the latest commit.

    Alternatively, trigger a new pipeline run manually.

  2. If the pipeline run is not yet finished, wait until it finishes.

  3. In the pipeline run details, click on the job that runs tests.

    Close to the end of its log, there should be a link to the test report at Allure TestOps. Make sure that it is present and works.

    The link to the launch is in the pipeline run log.

  4. In the test report at Allure TestOps, open a single test's results.

    At the bottom of the page, there should be a link back to the Azure pipeline. Make sure that it is present and works.

    The link to the Azure DevOps pipeline is at the bottom of the right pane.

2. Enable triggering Azure pipelines

On the Allure TestOps side, the integration with Azure DevOps needs to be configured on two levels.

First, the administrator specifies the Azure DevOps URL.

Then, any project's owner creates an authentication token in Azure devOps, adds it to Allure TestOps, and configures the job in Allure TestOps.

2.1. Specify Azure DevOps URL in Allure TestOps

  1. Log into Allure TestOps using an administrator account.

  2. Click on your avatar and go to Administration → Integrations.

  3. Click Add integration in the top right corner of the page.

  4. In the dialog that appears, select Azure DevOps Server.

  5. Fill in the fields:

    • Name — a name to help you recognize the integration, e.g., “Azure DevOps production”.
    • Endpoint — the URL of the Azure DevOps organization, e.g., “https://dev.azure.com/my-organization/”.

  6. Click Add integration.

2.2. Create a token in Azure DevOps

  1. In Azure DevOps, click on the User settings icon next to your avatar and select Personal access tokens.

  2. Click New Token.

  3. In the dialog that appears, fill in the fields:

    • Name — a name to help you recognize the token, e.g., “Token for Allure TestOps”.
    • Organization — the Azure DevOps organization that contains the project you want to integrate with Allure TestOps.
    • Expiration — how long the token must stay valid. After this date, the integration will stop working, and you will need to create a new token to continue using the integration.
    • Scopes — “Custom defined”.

  4. At the bottom of the dialog, click Show all scopes.

  5. Select the following scopes:

    • Build → Read & execute
    • Work Items → Read, write, & manage
    • Variable Groups → Read

  6. Click Create.

    The new token will become temporarily visible. Copy it.

    You will need this token on the next step.

2.3. Add the token to an Allure TestOps project

  1. In Allure TestOps, go to the project page.

  2. In the menu on the left, click Settings → Integrations.

  3. Under the Available integrations, find the Azure DevOps integration and click Add integration next to it.

  4. In the dialog that appears, fill in the fields:

    • Username (email) — the email that you use to log into Azure DevOps.
    • API token — the API token that you got on step 2.2.

    Check that the credentials are correct

    Click Test connection. After a few moments, a message should appear saying “Connection established”.

  5. Click Add integration to close the dialog and save the settings.

2.4. Configure the job in Allure TestOps

  1. In Allure TestOps, go to the project page.

  2. In the menu on the left, click Jobs.

    The page should contain a new job automatically added and configured during the launch on step 1.4.

  3. Click the job's three dots menu, then select Configure.

  4. In the dialog that appears, edit the fields:

    • Name — a name to help you recognize the job.
    • Build server — the name of the integration that you added on step 2.1.
    • Job can be used to run tests — if checked, users will be able to trigger this job from Allure TestOps.
    • Parameters — parameters that should be passed to Azure DevOps via environment variables, see Environment.

    The dialog allows to set job settings.

  5. Click Submit.

3. Parametrize jobs

Azure pipeline runs can accept variables. Allure TestOps integrates this feature with its own Environment concept, which lets you both set parameters for new jobs and see parameters set for jobs that were started outside Allure TestOps.

First, the administrator declares the required display names for parameters globally.

Then, any project's owner configures the parameters mapping in the project's settings and in the job's settings in Allure TestOps.

Finally, the variables must be also added to the pipeline in Azure DevOps.

If your project's repository has multiple branches, make sure to create a “Branch” environment variable in Allure TestOps and pass it to your job. This special name will instruct Azure DevOps which of the branches it should use.

3.1. Add global parameter names

  1. Log into Allure TestOps using an administrator account.

  2. Click on your avatar and go to Administration → Environment.

  3. For each parameter name that you want to add:

    1. Click Create.

    2. Enter the name of the new parameter.

    3. Click Submit.

    Environment variable names are added in the Administration settings.

3.2. Map parameters to environment variables

  1. In Allure TestOps, go to the project page.

  2. In the menu on the left, click Settings → Environment.

  3. For each parameter name that you want to use:

    1. Click Create if the parameter is not in the list. Otherwise, click the Edit icon next to its name.

    2. In the Mapping key field, specify the name of the environment variable.

    3. In the Environment variable field, select the global parameter name from step 3.1.

    4. Click Submit.

    Mapping environment variables is configured in the project settings.

3.3. Add parameters to the job

  1. In Allure TestOps, go to the project page.

  2. In the menu on the left, click Jobs.

  3. Find the job which you want to parametrize. Click the job's three dots menu, then select Configure.

    The job settings dialog will appear, containing the Parameters section.

  4. For each parameter that you want to add, click Add and fill in the fields:

    • Name — the name of the environment variable (same as the Mapping key from step 3.2).
    • Value — the default value that should be used unless overriden for a specific run.
    • Environment Variable — the environment variable from step 3.1.

    Multiple environment variables can be specified in the dialog.

  5. Click Submit.

3.4. Set the default values in pipeline

  1. In Azure DevOps, go to Pipelines and select the pipeline you need.

  2. In the top right corner, click Edit.

  3. In the top right, click the Variables button.

  4. For each parameter that you want to add, click + and fill in the fields:

    • Name — the name of the environment variable (same as the Mapping key from step 3.2 and Name from step 3.3).
    • Value — the default value that should be used unless overriden for a specific run (same as Value from step 3.3).
    • Keep this value secret — must not be checked.
    • Let users override this value when running this pipeline — must be checked.

    Click OK to finish editing the new variable.

  5. Click Save to apply the changes and close the dialog.

© Qameta Software Inc. All rights reserved.