Test results upload from Azure
Azure's upload workflow uses allurectl.
Foreword
This very article describes the basics – how to create a pipeline in Azure DevOps and describes in details on how to upload test results from Azure DevOps pipelines.
If you need to get more information on Azure pipelines, you need to refer to dedicated resources like official documentation pages.
Creation of a pipeline
- In your project go to pipelines.
- Then click special button to create a new pipeline.
Select the source for your code
- Select the VCS from the list
- Select the repository in VCS
You either need to use an existing yaml file with pipeline instructions or create a new one
Authenticate Azure's build job
For allurectl to send the test results to Allure TestOps server you need it to be authenticated.
To authenticate allurectl you need to perform 2 actions:
- Create API token on Allure TestOps side
- Provide API token to
allurectl
alongside with the information on Endpoint and used project where you will send the test results to.
API token
For a user on Allure TestOps side you need to create API token.
This API token will be used by allurectl
running in Azure pipeline to authenticate on Allure TestOps server.
- Log in to Allure TestOps with the account you are going to use to upload test results.
- Go to user's profile.
- In the section API tokens click a green button that says Create.
- Name your API token and click Submit. Always give meaningful names to all the configuration parameters.
Allure will randomly generate the token and show you in modal window.
- Copy the token and save it in a safe place as it cannot be retrieved by Allure TestOps means.
Environment variables
allurectl uses environment variables to get the information on Allure TestOps endpoint, project and API token.
To add the required variables you need to click Variables button when configuring your pipeline.
The following variables need to be added to allow the test results upload to ATO Allure Test:
ALLURE_ENDPOINT
- URL of Allure TestOps server.ALLURE_PROJECT_ID
- the ID of the Allure TestOps project.ALLURE_TOKEN
- the authentication token you created on the previous steps.ALLURE_JOB_RUN_ID
- this variable is a service one, it'll be used by allurectl for the proper mapping of the pipeline entities to Allure TestOps entities.
For the secrets you need to use "Keep this value secret" option.
ALLURE_JOB_RUN_ID service variable
This is a special service variable used by allurectl
.
ALLURE_JOB_RUN_ID needs to be set as follows:
- The value of
ALLURE_JOB_RUN_ID
needs to be empty - "Let users override this value when running this pipeline" needs to be checked
Other service variables
Please refer to the description of allurectl to add other variables you might need.
If you need to be able to update the value of the variable during the pipeline run, you need to use "Let users override this value when running this pipeline" option.
This also means, if you are using some environment variables in your tests like browser name, URL of a host to run the tests on etc., and you need to pass this information from Allure TestOps, you need to add these variables to the pipeline variables as well (see the example below).
That's it. Now, you can configure your build jobs to send test results to Allure TestOps server.
Pipelines
This section describes mandatory information and commands to be included into your pipeline for test results upload.
Pipeline example
In your Azure pipeline, you need to prepare all the data and tools for allurectl
to execute the tests and upload the test results to Allure TestOps server.
Here, is an example of pipeline (see the comments below the example):
<snip>
steps:
- task: [email protected]
displayName: 'Install pytest, install allure-pytest'
inputs:
targetType: 'inline'
script: |
pip install pytest allure-pytest
- task: [email protected]
displayName: 'Download and prepare allurectl'
inputs:
targetType: 'inline'
script: |
wget https://github.com/allure-framework/allurectl/releases/latest/download/allurectl_linux_386 -O ./allurectl
chmod +x ./allurectl
- script: |
./allurectl job-run plan --output-file $(ALLURE_TESTPLAN_PATH)
./allurectl watch -- pytest --alluredir=$(ALLURE_RESULTS)
displayName: 'running pytest tests'
env:
ALLURE_ENDPOINT: $(ALLURE_ENDPOINT)
ALLURE_PROJECT_ID: $(ALLURE_PROJECT_ID)
ALLURE_TOKEN: $(ALLURE_TOKEN)
ALLURE_TESTPLAN_PATH: $(ALLURE_TESTPLAN_PATH)
ALLURE_RESULTS: $(ALLURE_RESULTS)
ALLURE_LAUNCH_NAME: "$(Build.DefinitionName)-$(Build.BuildNumber)"
ALLURE_LAUNCH_TAGS: "$(Build.SourceBranchName), azure"
TEST_BRANCH: $(Build.SourceBranchName)
AZURE_BROWSER: $(AZURE_BROWSER)
AZURE_HOST: $(AZURE_HOST)
So, what you need to do in your pipeline:
- Prepare the base for your tests, i.e. install all the tools and dependencies (first
task: [email protected]
). - Download appropriate
allurectl
binary to your pipeline. See here the available ones., (secondtask: [email protected]
). - Make
allurectl
executable (secondtask: [email protected]
). - Execute
allurectl
to trigger your tests and upload the test results.- The construct
./allurectl job-run plan --output-file $(ALLURE_TESTPLAN_PATH)
is required to enable selective tests run on pipeline level (also depends on your test framework integration with Allure Framework). We advise defining the env variable$(ALLURE_TESTPLAN_PATH)
on pipeline's level. - The command
watch
is the preferred way to upload the test results to Allure TestOps. - Most convenient way to provide all the data for proper upload is to use environment variables, the upload command will have clearer format and it'll be easier to debug the pipeline.
- The construct
- Provide environment variables for the script with the test execution and test results upload.
ALLURE_ENDPOINT
,ALLURE_PROJECT_ID
,ALLURE_TOKEN
,ALLURE_RESULTS
are generally required to upload the test results data. These are processed byallurectl
automatically.ALLURE_TESTPLAN_PATH
is required for selective test run to work (also depends on Allure Framework adaptor capabilities). Processed byallurectl
automatically.ALLURE_LAUNCH_NAME
defines how launches will be displayed in Allure TestOps UI.ALLURE_LAUNCH_TAGS
provides additional metadata on a launch and also used in Allure TestOps UI as well as for searching and filtering of the launches.TEST_BRANCH
is environment variable used to collect the data on VCS branch name used to run the tests in the pipeline. This information is eventually used for triggering jobs from Allure TestOps side.AZURE_BROWSER
,AZURE_HOST
are env variables used to pass additional data to your tests. This information is also used to provide meta-data on launches and will be displayed in the UI.
The usage of the environment variables is described here.