TestOps logo
Try Testops

Integration with Jenkins CI

This integration is a specific case of Integrations with external systems.

Allure TestOps integration with Jenkins is bidirectional, this means you can do the following:

  • Upload test results from a Jenkins pipeline (Jenkins > Allure TestOps)
  • Test Connection with Jenkins (Allure TestOps > Jenkins)
    • Allure TestOps can check if provided credentials are valid
    • Allure TestOps cannot validate user's authorization in projects at this stage, i.e. if user tries triggering without required permissions, Unauthorized error will occur
  • Job Sync Operation
    • Allure TestOps can retrieve correct name and parameters for a pipeline
  • Job Suggest Operation
    • Allure TestOps is capable to retrieve the pipelines available for the current user
  • Job Run Operation
    • Allure TestOps can trigger a new build for a pipeline associated with configured Job (Allure TestOps > Jenkins)

Main Jenkins' upload workflow uses dedicated plug-in or allurectl. We provide basic templates ensuring the expected work with Allure TestOps on this page in the following sections.

Uploading the test results from Jenkins

Test results upload can be done using either Allure TestOps plugin or allurectl.

This section describes the plugin path.

Plugin installation

  1. Download allure-jenkins-x.xx.x.hpi plug-in from dl.qameta.io.
  2. In Jenkins' main page jump to Manage Jenkins >> Manage Plugins.

Jenkins manage plug-ins

  1. Jump to Advanced tab, click Choose File and select the allure-jenkins.hpi you downloaded in the step #1.

    Jenkins upload plug-ins

  2. Click Upload

  3. Jenkins will install the plug-in, and you will be able to proceed to further steps of the integration process.

Authenticate Jenkins' build jobs

Authentication token

For a user on Allure TestOps side you need to create API token. This API token will be used by Allure TestOps plugin for Jenkins to authenticate on Allure TestOps server.

  1. Log in to Allure TestOps with dedicated account you are going to use to upload test results. That could be your profile as well.
  2. Go to user's profile.

Allure go to profile

  1. In the section API tokens click a green button that says Create.
  2. Name your API token and click Submit. Always give meaningful names to all the configuration parameters.

Allure go to profile

Allure will randomly generate the token and show you in modal window.

  1. Copy the token and save it in a safe place as it cannot be retrieved by Allure TestOps means.

Add your Allure server instance information to Jenkins configuration

To upload the test results to a project in Allure TestOps the end user used to create API token must be explicitly added to the said project with at least Write permissions.

  1. Open Jenkins configuration page: Manage Jenkins > Configure System

  2. Navigate to Allure Servers section of the configuration page

Allure Server List

  1. Click Add Allure Server button and setup server location

Allure Server Add

  1. In Credentials section click Add button and create a credential with Kind parameter Kind = Secret text and paste the token you generated on the previous step.

Credentials Add

  1. Choose created credentials from the drop-down list Credentials and press Test Connection button

Allure Server Test

  1. Save Jenkins' configuration

That's it. Now, you can configure your build jobs to send test results to Allure TestOps server.

Configure a build job

The examples below show the following jobs configurations:

  1. Single Freestyle build job configuration.
  2. Single Pipeline build job configuration.

Single Freestyle build job configuration

  1. Open job configuration page
  2. Navigate to Build Environment section
  3. Enable Allure: upload results and set it up:
    1. Server is the label of Allure TestOps server you've assigned in Configure System of Jenkins.
    2. Project is a drop-down list that has all the project from Allure TestOps server. This will set the destination project where your test results will be stored.
    3. Launch name this will appear in Allure TestOps launches area, where all the tests results are stored, analysed and processed. If you use Jenkins' variables, then launch name will be dynamically changed accordingly to the build number.
    4. Launch tags tags for a particular job, on Allure TestOps side it will help you filtering the tests.
    5. Results path on the CI server, where you store the test results of this very build job.
    6. Advanced settings allow you altering the behaviour of the plug-in.
    7. On the Build stage add the tests execution command.

Freestyle Job

Single Pipeline build job configuration

To configure the pipeline's step for the test results upoad in the same way as we did it for a Freestyle job, you can use Jenkins' feature Pipeline syntax

Pipeline

In the new page Jenkins will open you need to fill the following to get the step for the test results upload to Allure TestOps.

  1. Select the step withAllureUpload
  2. Server is the label of Allure TestOps server you've assigned in Configure System of Jenkins.
  3. Project is a drop-down list that has all the project from Allure TestOps server. This will set the destination project where your test results will be stored.
  4. Launch name this will appear in Allure TestOps launches area, where all the tests results are stored, analysed and processed. If you use Jenkins' variables, then launch name will be dynamically changed accordingly to the build number.
  5. Launch tags tags for a particular job, on Allure TestOps side it will help you filtering the tests.
  6. Results path on the CI server, where you store the test results of this very build job.
  7. Advanced settings allow you altering the behaviour of the plug-in.
  8. Click Generate Pipeline Script

Pipeline step via Pipeline syntax helper

The pipeline syntax helper will generate something like following.

withAllureUpload(name: '${JOB_NAME} - #${BUILD_NUMBER}', projectId: '111', serverId: 'testing', tags: 'example, regular, importante', results: [[path: 'build/allure-results']]) {
    // some block
}

You need to replace // some block with the tests execution command.

Final pipeline will look like follows.

pipeline {
    agent any

    stages {
        stage('Checkout') {
            steps {
                git url: 'https://github.com/username/allure-example.git'
            }
        }
        stage('Build') {
            steps {
                withAllureUpload(name: '${JOB_NAME} - #${BUILD_NUMBER}', projectId: '111', serverId: 'testing', tags: 'example, regular, importante', results: [[path: 'build/allure-results']]) {
                    sh './gradlew clean test'
                }
            }
        }
    }
}

Single Pipeline build job configuration with few uploads

The configuration will work in the same way as for the pipeline with one upload.

When you use several uploads in a single pipeline, all the uploads will to to the same launch. This is considered as correct workflow accordingly to Allure TestOps workflows design.

Inside a launch test created in different stages for a single withAllureUpload routine can be grouped either based on tests' layer or based on different environment variables used for the tests execution. The processing and usage of the environment variables is described further on. 

pipeline {
    agent any

    stages {
        stage('checkout e2e') {
            steps {
                git branch: '${TESTS_BRANCH}', url: 'https://github.com/username/cm-pw-example-tests.git'
            }
        }
        stage('e2e tests - playwright') {
            steps {
            catchError(buildResult: 'UNSTABLE', message: 'It seems we have failing tests', stageResult: 'UNSTABLE') {    
            withAllureUpload(name: '${JOB_NAME} - #${BUILD_NUMBER}', projectId: '111', results: [[path: 'allure-results']], serverId: 'testing', tags: '${TESTS_BRANCH},api, e2e') {
                nodejs('node-16') {
                    sh "npm i"
                    sh "npm test"
                    }
                }
            }
          }
        }
        stage('checkout api') {
            steps {
                git branch: '${TESTS_BRANCH}', url: 'https://github.com/username/cm-demo-tests-api.git'
            }
        }
        stage('preps') {
            steps {
                sh 'chmod +x gradlew'
            }
        }
        stage('API tests exec') {
            steps {
            withAllureUpload(name: '${JOB_NAME} - #${BUILD_NUMBER}', projectId: '111', results: [[path: 'build/allure-results']], serverId: 'testing', tags: '${TESTS_BRANCH},api, e2e') {
                sh './gradlew clean test'    
            }
            }
        }
    }
}

Multi Pipeline build job (with child jobs)

Multi pipeline job with child jobs.

If you need to upload the test results from child jobs you ned to pass the value of the job run ID (ALLURE_JOB_RUN_ID) to child jobs, so those will be able to upload the data to the same launch which was created by the parent job.

Parent pipeline script allure-pipeline-parent

pipeline {
  agent any

  stages {
   stage('Build') {
     steps {
      withAllureLaunch(projectId: '1') {
        build job: 'allure-pipeline-child', parameters: [string(name: 'ALLURE_JOB_RUN_ID', value: env.ALLURE_JOB_RUN_ID)]
      } 
     }
   }
  }
}

Child pipeline script allure-pipeline-child

pipeline {
  agent any

  parameters {
    string(name: 'ALLURE_JOB_RUN_ID', defaultValue: '', description: 'Job run ID')
  }

  environment {
   ALLURE_JOB_RUN_CHILD = 'true'
  }

  stages {
   stage('Checkout') {
     steps {
      git 'https://github.com/eroshenkoam/allure-example.git'
     }
   }
   stage('Build') {
     steps {
      withAllureUpload(projectId: '1', results: [[path: 'build/allure-results']]) {
        sh "./gradlew clean test"
      } 
     }
   }
  }
}

Advanced parameters of ATO Allure TestOps plug-in for Jenkins CI

Parameters available in a pipeline

Advanced parameters alter the behaviour of the plug-in when the plug-in processes the test results in the scope of a pipeline.

Currently available parameters:

  1. Silent mode
  2. Index existing files
Silent mode

There could be situations like a network problem when Allure TestOps server becomes inaccessible for Jenkins CI. In this case the whole pipeline will fail as the plug-in will throw an exception, quit and tests won't be executed.

The parameter Silent mode allows to continue the execution of the tests even if Allure TestOps is inaccessible and the upload of the test results is not possible.

withAllureUpload(silent: true, name: 'jobname', projectId: '111', serverId: 'server',  tags: '') {
    // command to execute the tests
}

Having the said parameter set to true withAllureUpload routine will throw an exception, but it'll execute the command provided in its body for the tests execution. The test results then can be used to generate Allure Report or could be manually uploaded to ATO Allure TestOps server when it becomes available again.

Index existing files

There could be situations when for some reason the test results appear on the file system before withAllureUpload is executed in a pipeline. In this case withAllureUpload will ignore existing files and upload the files which appeared after withAllureUpload was executed by a pipeline.

To override ignoring of the existing test results and upload existing files you need to set the parameter indexExistingFiles to true as in the example below:

withAllureUpload(indexExistingFiles: true, name: '${JOB_NAME} - #${BUILD_NUMBER}', projectId: '', serverId: '', tags: '') {
    // some commands
}

Using allurectl in Jenkins pipelines

allurectl can be used in Jenkins pipelines as an alternative to the Allure TestOps plug-in for Jenkins when you are experiencing some troubles with the plug-in related to Jenkins internal timeouts (Jenkins unexpectedly terminates the plug-in and the test results upload is being interrupted).

allurectl will reuse the same authentication data and will require some variables to be added to your pipeline.

The token required for the authentication can be provided via withCredentials command.

The rest of the parameters required for allurectl are described in the section dedicated to allurectl

Example of a pipeline with allurectl

pipeline {
    agent any
    environment {
        ALLURE_ENDPOINT = 'https://allure.endpoint.xyz'
        ALLURE_PROJECT_ID = '777'
        ALLURE_RESULTS  = 'build/allure-results'
    }

    stages {
        stage('pull the tests code fom VCS') {
            steps {
                git branch: '${TESTS_BRANCH_NAME}', url: 'https://github.com/username/allure-example.git'
            }
        }
        stage('Running tests') {
            steps {
                withCredentials([string(credentialsId: 'jenkins_secret_name', variable: 'ALLURE_TOKEN')]) {
                    sh "wget https://github.com/allure-framework/allurectl/releases/latest/download/allurectl_linux_amd64 -O ./allurectl && chmod +x ./allurectl"
                    sh "echo 'cleaning up previous build data'"
                    sh "rm -r -f build/allure-results"
                    sh "echo 'making gradlew executable'"
                    sh "chmod +x ./gradlew"
                    sh "echo 'removing old testplan.json if any'"
                    sh "rm -f testplan.json || true"
                    sh "echo 'generating new testplan.json if data is received from Allure TestOps'"
                    sh "echo 'executing tests and sending test results to Allure TestOps'"
                    sh "./allurectl watch -- ./gradlew clean test"
                }
            }
        }
    }
}

Collecting the information about tests environment

Why?

It is quite a rare situation when tests run without any parameters which define different conditions to run the tests. These conditions could be for instance:

  • different hosts to run the tests on, e.g. QA server, staging server etc.
  • different browsers to run the tests on for web UI tests.
  • VCS branch to test changes before they appear in master branch.

When Allure TestOps collects this information, it will allow you starting the build jobs with the environment of your choice right from its UI.

How?

Add environment parameters to a Job on Jenkins side

Now, say by default our test will check the production site using its API but from time to time we might want to test QA version of the site or staging or whatever version we could have. This requires build jobs' parametrisation.

Adding parameters

Locate This project is parameterised in your build job settings and enable it.

We can create an environment variable called TESTS_ENDPOINT on Jenkins side and assign the value so the tests will run for the site assigned to this TESTS_ENDPOINT variable.

Env variable HOST

From now on, you can pass this environment variable to your tests using ${TESTS_ENDPOINT}. Of course you need to implement a bit of a code which will take the environment variable and use it in your tests.

Setting up Allure TestOps to process environment variables from Jenkins

Both, Allure TestOps plug-in or allurectl send whole set of environment variables from a pipeline to Allure TestOps. This data also includes added parameters.

Now, let's set up Allure TestOps project settings for this environments information processing.

  1. Proceed to the project settings
  2. Go to the Environment section
  3. Click Create
  4. Add env variable name (TESTS_ENDPOINT) to Mapping key
  5. Select global environment variable from the list (these are managed by Allure TestOps administrators)
  6. And Submit

Env variable TESTS_ENDPOINT

Add other variables you've created, and run the build job from Jenkins' UI again.

Env variable is shown in a launch

All other variables are to be added the same way. In the pictures above you can see, we added TESTS_BRANCH_NAME which we can use in Jenkins for VCS branch selection.

Select a branch to run on

This variable can be also processed by Allure TestOps and we'll most likely use it during forthcoming steps.

Trigger pipelines from Allure TestOps

To trigger pipelines on a CI system (this relates to any CI system in general) you need to have a specific integration added on a) global level b) on the current project level. Then this integration will be used in a Job settings.

This section covers

  1. Integration recap.
  2. Job settings.

The process of the configuration of the integrations with external systems is described here.

Integration recap

  1. Create (get/generate/you name it) the secrets (credentials) on an external system to use their API.
  2. Add the required integration on Allure TestOps global level (you need to be an administrator or have an administrator nearby).
  3. Add the required integration on the project level and provide credentials (secrets) you got on the step 1.

Global level

You need to have ROLE_ADMIN authority to configure Allure TestOps instance on the global level.

  1. Go to User menu > Administration > Integrations
  2. In the upper-right corner click Add integration
  3. Search for Jenkins
  4. Click on the Jenkins banner
  5. Provide the name of the integration
    • URL of Jenkins without http(s):// might be a good idea for the name
  6. Provide Jenkins' URL to Endpoint field
  7. Finalise by clicking Add integration.

Next you'll see the following screen:

Select a branch to run on

Here, you can test the connection (good idea!), update the parameters, or even delete the integration (which is bizarre considering you just added it).

Project level

On the level of a project you need to add the created integration to the project you work in, and provide the credentials (secrets) to authorize Allure TestOps to trigger the pipelines.

Jenkins credentials

To integrate with Jenkins we need a) user name b) token for the designated user.

You can either use your account, or alternatively you can create a new service account (up to you).

Jenkins account needs the permissions to start the builds.

  1. Go to the account configuration /user/{YOUR_USERNAME}/configure

User configuration

  1. Locate API token section
  2. Click Add new token
  3. Name the token
    • URL of ATO Allure TestOps instance without http(s):// might be a good idea for the name
  4. Click Generate
  5. Copy the created token

Copy token

Setup the integration in your project

You need to be a project owner to configure these settings.

  1. Go to the project's settings.
  2. Go to the Integration section
  3. Locate the Jenkins integration you want to add.
  4. Click Add integration.
  5. In the modal window enter the credentials – the username and token that we generated on the previous step.
  6. Test the connection
  7. Finalise the Integration settings by clicking on Add integration.

Fill the credentials

Now, we are ready to configure the Job.

Allure TestOps job settings

Job in Allure TestOps is an entity linked to a CI pipeline. The process of the test results upload is a job-run. All the parameters of a job-run are managed by Allure TestOps and CI plug-in (allurectl).

Pipeline has parameters (passed as env variables) which can be used to trigger a pipeline from Allure TestOps side. To do so, you need to finalise the settings for a Job and create an integration allowing you to trigger the jobs in a CI system.

To be able triggering the associated pipelines, you need to have the configured build server (just done on the previous step).

Also,

To start using a Job for creating Launches and triggering CI pipelines form Allure TestOps UI, you need to configure such a Job:

Job configuration

  1. Select correct CI server configured on Project level with correct access rights provided in CI.
  2. Mark the Job as available for running tests on it.
  3. Link the ENV variables received from the job to the global environment variables.
  4. Submit the changes.

Trigger a pipeline

Triggering of a pipeline (that'll create a launch as well) can be done in the following sections of Allure TestOps:

  • List of the test cases – select and use bulk operation Run.
  • List of test results – select and use bulk operation ReRun.
  • Test plans – Click Run button of a test plan.
  • Jobs – Click Run button of a test Job.

All the said actions will result in the modal window with the parameters of a launch Allure TestOps creates for such actions.

Configure launch parameters

Trigger a pipeline on an alternative environment

Alternative means, you want to use the parameters which differ from the ones used in a pipeline by default.

In the modal window showed above click Add environment

Configure launch parameters

Set the required values for the Environment variables you need to alter, and click Save environment.

You can add several sets of the environments, in this case Allure TestOps will create several pipeline executions with different parameters' set and all of them will upload the test results in the same launch.

© Qameta Software Inc. All rights reserved.