Integration with Jenkins CI
This integration is a specific case of Integrations with external systems.
Allure TestOps integration with Jenkins is bidirectional, this means you can do the following:
- Upload test results from a Jenkins pipeline (Jenkins > Allure TestOps)
- Jenkins upload workflow uses either allurectl or Jenkins plug-in to upload the test results.
- Test Connection with Jenkins (Allure TestOps > Jenkins)
- Allure TestOps can check if provided credentials are valid
- Allure TestOps cannot validate user's authorization in projects at this stage, i.e. if user tries triggering without required permissions, Unauthorized error will occur
- Job Sync Operation
- Allure TestOps can retrieve correct name and parameters for a pipeline
- Job Suggest Operation
- Allure TestOps is capable to retrieve the pipelines available for the current user
- Job Run Operation
- Allure TestOps can trigger a new build for a pipeline associated with configured Job (Allure TestOps > Jenkins)
Main Jenkins' upload workflow uses dedicated plug-in or allurectl. We provide basic templates ensuring the expected work with Allure TestOps on this page in the following sections.
Uploading the test results from Jenkins
Test results upload can be done using either Allure TestOps plugin or allurectl.
This section describes the plugin path.
Plugin installation
- Download
allure-jenkins-x.xx.x.hpi
plug-in fromdl.qameta.io
. - In Jenkins' main page jump to Manage Jenkins >> Manage Plugins.
Jump to Advanced tab, click Choose File and select the
allure-jenkins.hpi
you downloaded in the step #1.Click Upload
Jenkins will install the plug-in, and you will be able to proceed to further steps of the integration process.
Authenticate Jenkins' build jobs
Authentication token
For a user on Allure TestOps side you need to create API token. This API token will be used by Allure TestOps plugin for Jenkins to authenticate on Allure TestOps server.
- Log in to Allure TestOps with dedicated account you are going to use to upload test results. That could be your profile as well.
- Go to user's profile.
- In the section API tokens click a green button that says Create.
- Name your API token and click Submit. Always give meaningful names to all the configuration parameters.
Allure will randomly generate the token and show you in modal window.
- Copy the token and save it in a safe place as it cannot be retrieved by Allure TestOps means.
Add your Allure server instance information to Jenkins configuration
To upload the test results to a project in Allure TestOps the end user used to create API token must be explicitly added to the said project with at least
Write
permissions.
Open Jenkins configuration page: Manage Jenkins > Configure System
Navigate to Allure Servers section of the configuration page
- Click Add Allure Server button and setup server location
- In Credentials section click Add button and create a credential with Kind parameter Kind = Secret text and paste the token you generated on the previous step.
- Choose created credentials from the drop-down list Credentials and press Test Connection button
- Save Jenkins' configuration
That's it. Now, you can configure your build jobs to send test results to Allure TestOps server.
Configure a build job
The examples below show the following jobs configurations:
Single Freestyle build job configuration
- Open job configuration page
- Navigate to Build Environment section
- Enable Allure: upload results and set it up:
- Server is the label of Allure TestOps server you've assigned in Configure System of Jenkins.
- Project is a drop-down list that has all the project from Allure TestOps server. This will set the destination project where your test results will be stored.
- Launch name this will appear in Allure TestOps launches area, where all the tests results are stored, analysed and processed. If you use Jenkins' variables, then launch name will be dynamically changed accordingly to the build number.
- Launch tags tags for a particular job, on Allure TestOps side it will help you filtering the tests.
- Results path on the CI server, where you store the test results of this very build job.
- Advanced settings allow you altering the behaviour of the plug-in.
- On the Build stage add the tests execution command.
Single Pipeline build job configuration
To configure the pipeline's step for the test results upoad in the same way as we did it for a Freestyle job, you can use Jenkins' feature Pipeline syntax
In the new page Jenkins will open you need to fill the following to get the step for the test results upload to Allure TestOps.
- Select the step
withAllureUpload
- Server is the label of Allure TestOps server you've assigned in Configure System of Jenkins.
- Project is a drop-down list that has all the project from Allure TestOps server. This will set the destination project where your test results will be stored.
- Launch name this will appear in Allure TestOps launches area, where all the tests results are stored, analysed and processed. If you use Jenkins' variables, then launch name will be dynamically changed accordingly to the build number.
- Launch tags tags for a particular job, on Allure TestOps side it will help you filtering the tests.
- Results path on the CI server, where you store the test results of this very build job.
- Advanced settings allow you altering the behaviour of the plug-in.
- Click Generate Pipeline Script
The pipeline syntax helper will generate something like following.
withAllureUpload(name: '${JOB_NAME} - #${BUILD_NUMBER}', projectId: '111', serverId: 'testing', tags: 'example, regular, importante', results: [[path: 'build/allure-results']]) {
// some block
}
You need to replace // some block
with the tests execution command.
Final pipeline will look like follows.
pipeline {
agent any
stages {
stage('Checkout') {
steps {
git url: 'https://github.com/username/allure-example.git'
}
}
stage('Build') {
steps {
withAllureUpload(name: '${JOB_NAME} - #${BUILD_NUMBER}', projectId: '111', serverId: 'testing', tags: 'example, regular, importante', results: [[path: 'build/allure-results']]) {
sh './gradlew clean test'
}
}
}
}
}
Single Pipeline build job configuration with few uploads
The configuration will work in the same way as for the pipeline with one upload.
When you use several uploads in a single pipeline, all the uploads will to to the same launch. This is considered as correct workflow accordingly to Allure TestOps workflows design.
Inside a launch test created in different stages for a single withAllureUpload
routine can be grouped either based on tests' layer or based on different environment variables used for the tests execution. The processing and usage of the environment variables is described further on.
pipeline {
agent any
stages {
stage('checkout e2e') {
steps {
git branch: '${TESTS_BRANCH}', url: 'https://github.com/username/cm-pw-example-tests.git'
}
}
stage('e2e tests - playwright') {
steps {
catchError(buildResult: 'UNSTABLE', message: 'It seems we have failing tests', stageResult: 'UNSTABLE') {
withAllureUpload(name: '${JOB_NAME} - #${BUILD_NUMBER}', projectId: '111', results: [[path: 'allure-results']], serverId: 'testing', tags: '${TESTS_BRANCH},api, e2e') {
nodejs('node-16') {
sh "npm i"
sh "npm test"
}
}
}
}
}
stage('checkout api') {
steps {
git branch: '${TESTS_BRANCH}', url: 'https://github.com/username/cm-demo-tests-api.git'
}
}
stage('preps') {
steps {
sh 'chmod +x gradlew'
}
}
stage('API tests exec') {
steps {
withAllureUpload(name: '${JOB_NAME} - #${BUILD_NUMBER}', projectId: '111', results: [[path: 'build/allure-results']], serverId: 'testing', tags: '${TESTS_BRANCH},api, e2e') {
sh './gradlew clean test'
}
}
}
}
}
Multi Pipeline build job (with child jobs)
Multi pipeline job with child jobs.
If you need to upload the test results from child jobs you ned to pass the value of the job run ID (ALLURE_JOB_RUN_ID
) to child jobs, so those will be able to upload the data to the same launch which was created by the parent job.
Parent pipeline script allure-pipeline-parent
pipeline {
agent any
stages {
stage('Build') {
steps {
withAllureLaunch(projectId: '1') {
build job: 'allure-pipeline-child', parameters: [string(name: 'ALLURE_JOB_RUN_ID', value: env.ALLURE_JOB_RUN_ID)]
}
}
}
}
}
Child pipeline script allure-pipeline-child
pipeline {
agent any
parameters {
string(name: 'ALLURE_JOB_RUN_ID', defaultValue: '', description: 'Job run ID')
}
environment {
ALLURE_JOB_RUN_CHILD = 'true'
}
stages {
stage('Checkout') {
steps {
git 'https://github.com/eroshenkoam/allure-example.git'
}
}
stage('Build') {
steps {
withAllureUpload(projectId: '1', results: [[path: 'build/allure-results']]) {
sh "./gradlew clean test"
}
}
}
}
}
Advanced parameters of ATO Allure TestOps plug-in for Jenkins CI
Parameters available in a pipeline
Advanced parameters alter the behaviour of the plug-in when the plug-in processes the test results in the scope of a pipeline.
Currently available parameters:
- Silent mode
- Index existing files
Silent mode
There could be situations like a network problem when Allure TestOps server becomes inaccessible for Jenkins CI. In this case the whole pipeline will fail as the plug-in will throw an exception, quit and tests won't be executed.
The parameter Silent mode allows to continue the execution of the tests even if Allure TestOps is inaccessible and the upload of the test results is not possible.
withAllureUpload(silent: true, name: 'jobname', projectId: '111', serverId: 'server', tags: '') {
// command to execute the tests
}
Having the said parameter set to true
withAllureUpload
routine will throw an exception, but it'll execute the command provided in its body for the tests execution. The test results then can be used to generate Allure Report or could be manually uploaded to ATO Allure TestOps server when it becomes available again.
Index existing files
There could be situations when for some reason the test results appear on the file system before withAllureUpload
is executed in a pipeline. In this case withAllureUpload
will ignore existing files and upload the files which appeared after withAllureUpload
was executed by a pipeline.
To override ignoring of the existing test results and upload existing files you need to set the parameter indexExistingFiles
to true
as in the example below:
withAllureUpload(indexExistingFiles: true, name: '${JOB_NAME} - #${BUILD_NUMBER}', projectId: '', serverId: '', tags: '') {
// some commands
}
Using allurectl in Jenkins pipelines
allurectl
can be used in Jenkins pipelines as an alternative to the Allure TestOps plug-in for Jenkins when you are experiencing some troubles with the plug-in related to Jenkins internal timeouts (Jenkins unexpectedly terminates the plug-in and the test results upload is being interrupted).
allurectl
will reuse the same authentication data and will require some variables to be added to your pipeline.
The token required for the authentication can be provided via withCredentials
command.
The rest of the parameters required for allurectl
are described in the section dedicated to allurectl
Example of a pipeline with allurectl
pipeline {
agent any
environment {
ALLURE_ENDPOINT = 'https://allure.endpoint.xyz'
ALLURE_PROJECT_ID = '777'
ALLURE_RESULTS = 'build/allure-results'
}
stages {
stage('pull the tests code fom VCS') {
steps {
git branch: '${TESTS_BRANCH_NAME}', url: 'https://github.com/username/allure-example.git'
}
}
stage('Running tests') {
steps {
withCredentials([string(credentialsId: 'jenkins_secret_name', variable: 'ALLURE_TOKEN')]) {
sh "wget https://github.com/allure-framework/allurectl/releases/latest/download/allurectl_linux_amd64 -O ./allurectl && chmod +x ./allurectl"
sh "echo 'cleaning up previous build data'"
sh "rm -r -f build/allure-results"
sh "echo 'making gradlew executable'"
sh "chmod +x ./gradlew"
sh "echo 'removing old testplan.json if any'"
sh "rm -f testplan.json || true"
sh "echo 'generating new testplan.json if data is received from Allure TestOps'"
sh "echo 'executing tests and sending test results to Allure TestOps'"
sh "./allurectl watch -- ./gradlew clean test"
}
}
}
}
}
Collecting the information about tests environment
Why?
It is quite a rare situation when tests run without any parameters which define different conditions to run the tests. These conditions could be for instance:
- different hosts to run the tests on, e.g. QA server, staging server etc.
- different browsers to run the tests on for web UI tests.
- VCS branch to test changes before they appear in master branch.
When Allure TestOps collects this information, it will allow you starting the build jobs with the environment of your choice right from its UI.
How?
Add environment parameters to a Job on Jenkins side
Now, say by default our test will check the production site using its API but from time to time we might want to test QA version of the site or staging or whatever version we could have. This requires build jobs' parametrisation.
Adding parameters
Locate This project is parameterised in your build job settings and enable it.
We can create an environment variable called TESTS_ENDPOINT on Jenkins side and assign the value so the tests will run for the site assigned to this TESTS_ENDPOINT variable.
From now on, you can pass this environment variable to your tests using ${TESTS_ENDPOINT}
. Of course you need to implement a bit of a code which will take the environment variable and use it in your tests.
Setting up Allure TestOps to process environment variables from Jenkins
Both, Allure TestOps plug-in or allurectl send whole set of environment variables from a pipeline to Allure TestOps. This data also includes added parameters.
Now, let's set up Allure TestOps project settings for this environments information processing.
- Proceed to the project settings
- Go to the Environment section
- Click Create
- Add env variable name (
TESTS_ENDPOINT
) to Mapping key - Select global environment variable from the list (these are managed by Allure TestOps administrators)
- And Submit
Add other variables you've created, and run the build job from Jenkins' UI again.
All other variables are to be added the same way. In the pictures above you can see, we added TESTS_BRANCH_NAME
which we can use in Jenkins for VCS branch selection.
This variable can be also processed by Allure TestOps and we'll most likely use it during forthcoming steps.
Trigger pipelines from Allure TestOps
To trigger pipelines on a CI system (this relates to any CI system in general) you need to have a specific integration added on a) global level b) on the current project level. Then this integration will be used in a Job settings.
This section covers
- Integration recap.
- Job settings.
The process of the configuration of the integrations with external systems is described here.
Integration recap
- Create (get/generate/you name it) the secrets (credentials) on an external system to use their API.
- Add the required integration on Allure TestOps global level (you need to be an administrator or have an administrator nearby).
- Add the required integration on the project level and provide credentials (secrets) you got on the step 1.
Global level
You need to have
ROLE_ADMIN
authority to configure Allure TestOps instance on the global level.
- Go to User menu > Administration > Integrations
- In the upper-right corner click Add integration
- Search for Jenkins
- Click on the Jenkins banner
- Provide the name of the integration
- URL of Jenkins without
http(s)://
might be a good idea for the name
- URL of Jenkins without
- Provide Jenkins' URL to Endpoint field
- Finalise by clicking Add integration.
Next you'll see the following screen:
Here, you can test the connection (good idea!), update the parameters, or even delete the integration (which is bizarre considering you just added it).
Project level
On the level of a project you need to add the created integration to the project you work in, and provide the credentials (secrets) to authorize Allure TestOps to trigger the pipelines.
Jenkins credentials
To integrate with Jenkins we need a) user name b) token for the designated user.
You can either use your account, or alternatively you can create a new service account (up to you).
Jenkins account needs the permissions to start the builds.
- Go to the account configuration
/user/{YOUR_USERNAME}/configure
- Locate API token section
- Click Add new token
- Name the token
- URL of ATO Allure TestOps instance without
http(s)://
might be a good idea for the name
- URL of ATO Allure TestOps instance without
- Click Generate
- Copy the created token
Setup the integration in your project
You need to be a project owner to configure these settings.
- Go to the project's settings.
- Go to the Integration section
- Locate the Jenkins integration you want to add.
- Click Add integration.
- In the modal window enter the credentials – the username and token that we generated on the previous step.
- Test the connection
- Finalise the Integration settings by clicking on Add integration.
Now, we are ready to configure the Job.
Allure TestOps job settings
Job in Allure TestOps is an entity linked to a CI pipeline. The process of the test results upload is a job-run. All the parameters of a job-run are managed by Allure TestOps and CI plug-in (allurectl).
Pipeline has parameters (passed as env variables) which can be used to trigger a pipeline from Allure TestOps side. To do so, you need to finalise the settings for a Job and create an integration allowing you to trigger the jobs in a CI system.
To be able triggering the associated pipelines, you need to have the configured build server (just done on the previous step).
Also,
To start using a Job for creating Launches and triggering CI pipelines form Allure TestOps UI, you need to configure such a Job:
- Select correct CI server configured on Project level with correct access rights provided in CI.
- Mark the Job as available for running tests on it.
- Link the ENV variables received from the job to the global environment variables.
- Submit the changes.
Trigger a pipeline
Triggering of a pipeline (that'll create a launch as well) can be done in the following sections of Allure TestOps:
- List of the test cases – select and use bulk operation Run.
- List of test results – select and use bulk operation ReRun.
- Test plans – Click Run button of a test plan.
- Jobs – Click Run button of a test Job.
All the said actions will result in the modal window with the parameters of a launch Allure TestOps creates for such actions.
Trigger a pipeline on an alternative environment
Alternative means, you want to use the parameters which differ from the ones used in a pipeline by default.
In the modal window showed above click Add environment
Set the required values for the Environment variables you need to alter, and click Save environment.
You can add several sets of the environments, in this case Allure TestOps will create several pipeline executions with different parameters' set and all of them will upload the test results in the same launch.