Test results upload
The aim of this chapter is to describe the general workflow of test results upload to Allure Testops.
Test Results format
Your test generate some results containing information on the tests execution status. These test results have different formats depending on a test framework used.
Allure Report aka Allure Framework test results format is a universal one, and it has a lot of nice features allowing you to supply additional (meta) information for your tests. Using the integration with Allure Report allows you building nice reports with Allure Report generator, and use the same test results files to upload the reports to Allure Testops.
These are the test results formats supported on Allure Testops side, but again, the recommended format is Allure Report ver 2 as it allows provide additional information.
Name of the format | format of files | Comments |
---|---|---|
allure1 | *.xml |
old Allure Framework's test results format, still exist can be used by 3P adaptors |
allure2 | *-result.json *-container.json *-attachment.[ext] |
actual Allure Report test results format |
cucumber json | *.json |
JSON format of cucumber tests |
jgiven | test results | |
junit | *.xml |
JUnit XML files |
trx files | *trx |
Visual Studio test result files. Dinosaurs |
xctest | folder Report.xcresul |
xcode 11 supported without transformation. Newer versions need a converter. |
xunit | *.xml |
test results for xUnit test framework |
Test result upload options
Here are the options how can you upload the test results.
- CI plug-in or allurectl from a CI pipeline.
- e.g. Allure Testops plug-in for Jenkins
- allurectl for the rest of CI systems with scripted pipelines like GitLab, CircelCI, GitHub etc.
- Manual upload of test result via UI. Use upload button in Launches section or just drag these to Allure Testops UI in launches section.
- Use Allure: Upload results menu item in JetBrains' IDE. Check here for the details.
General process of the upload for the test results from CI
Important to understand – when it comes to the sending of the test results to Allure Testops, the tests do not send the test results.
Tests are doing what they're supposed to do – checking your application as you've programmed them. Full stop.
Here is the description on how your test results end up in Allure Testops, please check the picture below and read the description.
- CI's pipeline triggers Allure Testops' agent which ultimately is responsible for the upload of the test results.
- In case of GitLab, GitHub, CircleCI this will be allurectl.
- In case of Jenkins this will be the Allure Testops plugin for Jenkins.
- Allure Testops agent gets the connection params from the settings, receives data from Allure Testops (if the execution is initiated from Allure Testops) and triggers the tests execution.
- Allure Testops agent starts watching the allure-results folder.
- Your code executes the tests.
- Tests' logic uses test framework.
- Test's logic uses allure framework which listens to the events of test framework.
- Allure framework saves the events from a test framework to test results files.
- Allure Testops agent sends the test results in async mode to Allure Testops.
CI pipeline preparation
Despite the CI you are using, there are some steps you need to perform for all of them:
- Authenticate and authorize your pipeline to upload the test results
Authenticate your pipeline
This step requires the generation of API key, which is to be provided to an Allure Testops' agent.
- Log in to Allure Testops you are going to use to upload test results.
- Go to user's profile.
- In the section API tokens click a button saying + Create.
- Name your API token and click Submit.
Allure will randomly generate the token and show you in modal window.
- Copy the token and save it in a safe place as it cannot be retrieved by Allure Testops means.
- This string is to be provided to CI pipeline later on.
Authorization
To upload test results to a project the user whose API token is used in a pipeline, needs to be added to this project with Write access rights.
Despite all users of an Allure Testops instance can see (read) public project, all users must be explicitly added to a Public project with Project Write
permission to be able to upload the test results.
If you are trying to upload test results using API token of a user which is not added to the project you are working with, then no data won't be uploaded.
Supported CI servers
The table below shows the information on the Allure Testops agent type needed for different CI servers.
Basically, the test results upload is possible from any CI which supports external tool usage, i.e. download allurectl using wget/curl during the execution and then using allurectl to trigger the tests and upload the test results.
Here is a list of CI servers already tested and widely used with Allure Testops.
Integration | Tool to be used |
---|---|
AWS code pipelines | allurectl |
Azure DevOps | allurectl |
Bamboo | Allure Testops plug-in for Bamboo |
CircleCI (WIP) | allurectl |
GitHub | allurectl |
GitLab | allurectl |
Jenkins | Allure Testops plug-in for Jenkins or allurectl |
TeamCity | Allure Testops plug-in for TeamCity |
Tekton | allurectl |