TestOps logo
Try TestOps

Integration with Jenkins

This page describes how to set up both Allure TestOps and Jenkins in a project that uses Jenkins to run tests.

Like all other CI integrations for Allure TestOps, this involves communication in two directions.

  • The Jenkins build will send its status and test results to Allure TestOps.
  • Allure TestOps will use Jenkins API to trigger new pipelines, optionally passing a test plan.

Once a pipeline run stops, the corresponding launch in Allure TestOps may be closed either manually or automatically, according to the Auto close policy.

To enable Allure TestOps support in your Jenkins project:

  1. install the plugin for Jenkins,
  2. enable sending data from Jenkins,
  3. enable triggering Jenkins builds,
  4. parametrize jobs (if necessary).

1. Install the plugin for Jenkins

The recommended method of installing the Allure TestOps plugin is by specifying its URL. However, if your Jenkins configuration does not have access to the internet, you can install the plugin by uploading an HPI file to Jenkins manually. The selected installation method does not affect the functionality of the plugin.

Install from the URL

  1. Visit the allure-testops-jenkins directory at dl.qameta.io and find the latest version of the plugin. Copy the URL of the HPI file.

Please take into account, starting from the 15th Feb 2024 there are 2 versions of the plugin

  1. In Jenkins, go to Manage Jenkins → Manage Plugins → Advanced settings.

  2. Under the Deploy Plugin section, paste the URL of the HPI file.

  3. Click the Deploy button to confirm the installation.

    On the Download progress page, wait until each status is “Success”.

Install from a file

  1. Visit the allure-testops-jenkins directory at dl.qameta.io and find the latest version of the plugin. Save the HPI file to a local directory on your device.

  2. In Jenkins, go to Manage Jenkins → Manage Plugins → Advanced settings.

  3. Under the Deploy Plugin section, click Browse and select the HPI archive from your device.

  4. Click the Deploy button to confirm the installation.

    On the Download progress page, wait until each status is “Success”.

2. Enable sending data from Jenkins

This section describes the second part of the two-way communication: sending job statuses and test results from Jenkins back to Allure TestOps. To configure this, any Allure TestOps user can generate an authentication token and add it to Jenkins, so that Jenkins can perform actions with that user's permissions.

The Jenkins job itself also needs to be modified so that it includes communicating with the server. The plugin supports doing so for both freestyle projects and pipeline projects.

After configuring, run and check the job to make sure that everything works so far.

2.1. Create a token in Allure TestOps

  1. In Allure TestOps, click on your avatar and go to Your profile.

  2. Under the API tokens section, click Create.

  3. Enter a Token name (e.g., “Token for Jenkins”), then click Submit.

  4. In the dialog that appears, click the Copy icon to copy the token into clipboard.

    You will need this token on the next step.

2.2. Specify the server and token in Jenkins

  1. In Jenkins, go to Manage Jenkins → System.

  2. Under the Allure section, click Add Allure Server → Allure Server.

  3. Fill in the fields:

    • ID — a name to help you recognize the Allure TestOps server.
    • Endpoint — the URL of the Allure TestOps server.

  4. Near the Credentials field, click Add → Jenkins.

  5. In the dialog that appears, fill in the fields:

    • Kind — “Secret text”.
    • Secret — the API token that you got on step 2.1.
    • ID — a name to help you recognize the credentials, e.g., “allure-credentials”.

    Click Add to close the dialog.

  6. In the Credentials dropdown, select the ID of the stored credentials, e.g., “allure-credentials”.

  7. If your Allure TestOps server uses a self-signed SSL certificate, uncheck the Enable certificate validation checkbox. Otherwise, leave it checked.

    Check that everything so far is correct

    Click Test connection. After a few moments, a message should appear saying “Successfully authorized as ⟨USERNAME⟩”.

  8. Click Advanced for additional parameters:

    • Threads count — maximum number of threads for uploading files to the server.
    • Batch size — maximum number of files to upload to the server at a time (by one thread).
    • Results depth — number of subdirectory levels in the test results directory to scan and upload. “0” means no subdirectories will be uploaded.
    • Process interval — minimum interval (in seconds) to wait between sending batches of files (by one thread).
    • Indexing interval — interval (in seconds) to wait between scans of the test results directory.
    • Delay interval — minimum interval (in seconds) to wait before uploading a newly created file (to avoid uploading half-written files).
    • Delay before upload finish — interval (in seconds) to wait after the job is finished before assuming that no other files will appear in the directory. This is to avoid issues with some test runners that may spawn file writers in new processes.

  9. Click Save at the bottom of the page.

2.3a. Modify a freestyle project

In Jenkins, a “freestyle project” is a build configuration for which you add and edit steps via the web interface. Once the Allure TestOps plugin is installed, you can use the same web interface to enable and configure automatic upload of test results to the Allure TestOps server.

  1. In Jenkins, select a job you want to enable upload for.

  2. In the menu on the left, click Configure.

  3. Under the Build Environment section, check the Allure: upload results checkbox.

  4. Fill in the fields:

    • Server — the name of the Allure TestOps server that you specified on step 2.2.
    • Project — the ID of the Allure TestOps project.
    • Launch name — a template for naming test launches. You can use any Jenkins' environment variables here (see the /env-vars.html page on your Jenkins installation for the full list).
    • Launch tags — comma-separated list of tags that should be assigned to the test launch.

  5. Click Add results → Results.

  6. In the Results → Path field, specify the path to the test results directory, e.g., “build/allure-results”.

    If you have multiple build steps generating test results into multiple directories, use the Add button to specify more paths.

    Please note that the Jenkins plugin currently does not support wildcards and other patterns in the Path field. This will be improved in the upcoming versions.

  7. Click Advanced for additional job parameters:

    • Silent mode — if checked, a connection failure when attempting to upload test results will not be treated as a job failure.

      In general, it is only recommended to enable this mode when you have other means of viewing the test results, e.g., you have the Allure Report plugin for Jenkins installed as well.

    • Index Existing Files — if checked, Allure TestOps will not upload any files that were not created or modified by the job.

  8. Click Save.

2.3b. Modify a Jenkins Pipeline script

If you store the job in a Jenkins Pipeline script, the Allure TestOps plugin both adds the withAllureUpload() block and a web interface for generating a snippet which uses it.

  1. In Jenkins, select a job you want to enable upload for.

  2. In the menu on the left, click Pipeline Syntax.

    An interface for generating code snippets for Jenkins Pipeline will open.

  3. In the Sample Step dropdown list, select “withAllureUpload: Allure: upload results”.

  4. Fill in the fields:

    • Server — the name of the Allure TestOps server that you specified on step 2.2.
    • Project — the ID of the Allure TestOps project.
    • Launch name — a template for naming test launches. You can use any Jenkins' environment variables here (see the /env-vars.html page on your Jenkins installation for the full list).
    • Launch tags — comma-separated list of tags that should be assigned to the test launch.

  5. Click Add results → Results.

  6. In the Results → Path field, specify the path to the test results directory, e.g., “build/allure-results”.

    If you have multiple build steps generating test results into multiple directories, use the Add button to specify more paths.

    Please note that the Jenkins plugin currently does not support wildcards and other patterns in the Path field. This will be improved in the upcoming versions.

  7. Click Advanced for additional job parameters:

    • Silent mode — if checked, a connection failure when attempting to upload test results will not be treated as a job failure.

      In general, it is only recommended to enable this mode when you have other means of viewing the test results, e.g., you have the Allure Report plugin for Jenkins installed as well.

    • Index Existing Files — if checked, Allure TestOps will upload all files found in the target directory, even if they were not updated during this run.

  8. Click Generate Pipeline Script.

    In the area below the button, the generated snippet will appear, for example:

    withAllureUpload(name: '${JOB_NAME} - #${BUILD_NUMBER}', projectId: '1', results: [[path: 'build/allure-results']], serverId: 'Allure TestOps production', tags: '') {
    // some block
}

  9. Copy the generated snippet into clipboard to use later.

  10. Open the Groovy script that you use for the project. It may be:

    • the script in the text box in Configure → Pipeline,
    • the Jenkinsfile under the project's version control system (see the official documentation).

  11. Find the command (or the set of commands) that runs the tests and wrap them into the withAllureUpload() block from the generated snippet.

    If you use the Declarative Pipeline syntax, the withAllureUpload() block must be placed inside a steps block. For example:

    pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                git 'https://github.com/eroshenkoam/allure-example.git'
                withAllureUpload(name: '${JOB_NAME} - #${BUILD_NUMBER}', projectId: '1', results: [[path: 'build/allure-results']], serverId: 'Allure TestOps production', tags: '') {
                    sh './gradlew clean test'
                }
            }
        }
    }
}

    If you use the Scripted Pipeline syntax, the withAllureUpload() block must be placed inside a stage() block. For example:

    node {
    stage('Build') {
        git 'https://github.com/eroshenkoam/allure-example.git'
        withAllureUpload(name: '${JOB_NAME} - #${BUILD_NUMBER}', projectId: '1', results: [[path: 'build/allure-results']], serverId: 'Allure TestOps production', tags: '') {
            sh './gradlew clean test'
        }
    }
}

  12. Save the script. In the case of a Jenkinsfile, make sure to commit the changes to the repository.

2.4. Run and check the job

  1. In Jenkins, select the job for which you configured the Allure TestOps integration.

  2. Click Build Now.

    Wait until the build finishes.

  3. In the Build History, click on the latest build.

  4. Both in the menu and at the bottom of the page, there should be links to the test report at Allure TestOps. Make sure that it is present and works.

    The link to the launch is in the pipeline run log.

  5. In the test report at Allure TestOps, open a single test's results.

    At the bottom of the page, there should be a link back to Jenkins. Make sure that it is present and works.

    The link to the Jenkins pipeline is on the bottom.

3. Enable triggering Jenkins builds

Integration with the Jenkins server needs to be configured on two levels.

First, the administrator specifies the URL of Jenkins server.

Then, any project's owner creates an authentication token in Jenkins, adds it to Allure TestOps, and configures the job in Allure TestOps.

3.1. Specify Jenkins server in Allure TestOps

  1. Log into Allure TestOps using an administrator account.

  2. Click on your avatar and go to Administration → Integrations.

  3. Click Add integration in the top right corner of the page.

  4. In the dialog that appears, select Jenkins.

  5. Fill in the fields:

    • Name — a name to help you recognize the Jenkins server, e.g., “Jenkins production”.
    • Endpoint — the URL of the Jenkins server, e.g., “https://jenkins.example.com/”.

  6. If your Jenkins server uses a self-signed SSL certificate, check the Disable certificate validation checkbox.

  7. Click Add integration.

3.2. Create a token in Jenkins

  1. In Jenkins, click on your username and go to Security.

  2. Under the API Token section, click Add new Token.

  3. Enter a name (e.g., “Token for Allure TestOps”), then click Generate.

  4. In the dialog that appears, click the Copy icon to copy the token into clipboard.

    You will need this token on the next step.

3.3. Add the token to an Allure TestOps project

  1. In Allure TestOps, go to a project page.

  2. In the menu on the left, click Settings → Integrations.

  3. Under the Available integrations, find the Jenkins integration and click Add integration next to it.

  4. In the dialog that appears, fill in the fields:

    • Username — the Jenkins username (from the profile page in Jenkins).
    • API token — the API token that you got on step 3.2.

    Check that the credentials are correct

    Click Test connection. After a few moments, a message should appear saying “Connection established”.

  5. Click Add integration to close the dialog and save the settings.

3.4. Configure the job in Allure TestOps

  1. In Allure TestOps, go to the project page.

  2. In the menu on the left, click Jobs.

    The page should contain a new job automatically added and configured during the launch on step 2.4.

  3. Click the job's three dots menu, then select Configure.

  4. In the dialog that appears, edit the fields:

    • Name — a name to help you recognize the job.
    • Build server — the name of the integration that you added on step 3.1.
    • Job can be used to run tests — if checked, users will be able to trigger this job from Allure TestOps.
    • Parameters — parameters that should be passed to Jenkins via environment variables, see Environment.

    The dialog allows to set job settings.

  5. Click Submit.

4. Parametrize jobs

Jenkins supports parametrized builds. Allure TestOps integrates this feature with its own Environment concept, which lets you both set parameters for new jobs and see parameters set for jobs that were started outside Allure TestOps.

In Jenkins, the project's owner adds variables and their default values.

In Allure TestOps, the administrator declares the required display names for parameters globally, then the project's owner configures the parameters mapping both in the project's settings and in the job's settings.

If your project's repository has multiple branches, make sure to create a “Branch” environment variable in Allure TestOps and pass it to your job. This special name will instruct Jenkins which of the branches it should use.

4.1. Set the default values in Jenkins

  1. In Jenkins, select the job for which you configured the Allure TestOps integration.

  2. In the menu on the left, click Configure.

  3. Under the General section, check the This project is parameterized checkbox.

  4. For each parameter that you want to add, click Add Parameter → String Parameter and fill in the fields:

    • Name — the name of the variable.
    • Default Value — the default value that should be used unless overriden for a specific launch.

    Click Save to save the new parameters.

    The string parameter can be added to a Jenkins pipeline.

4.2. Add global parameter names in Allure TestOps

  1. Log into Allure TestOps using an administrator account.

  2. Click on your avatar and go to Administration → Environment.

  3. For each parameter name that you want to add, click Create, enter the name of the new parameter and click Submit.

    Note that this name can be a human-readable label and does not have to be equal to the name from step 4.1.

    Environment variable names are added in the Administration settings.

4.3. Map parameters to environment variables in Allure TestOps

  1. In Allure TestOps, go to the project page.

  2. In the menu on the left, click Settings → Environment.

  3. For each parameter name that you want to use:

    1. Click Create if the parameter is not in the list. Otherwise, click the Edit icon next to its name.

    2. In the Mapping key field, specify the name of the environment variable name from step 4.1.

    3. In the Environment variable field, select the global parameter name from step 4.2.

    4. Click Submit.

    Mapping environment variables is configured in the project settings.

4.4. Load the parameters into the Allure TestOps job

  1. In Allure TestOps, go to the project page.

  2. In the menu on the left, click Jobs.

  3. Find the job which you want to parametrize. Click the Refresh button next to it.

    The parameter names and their default values should appear in the job description.

    The parameters list is under the job name.

  4. In the job's three dots menu, click Configure.

    The job settings dialog will appear, containing the Parameters section.

  5. For each parameter that you want to use, select the corresponding Environment variable from step 4.2.

    Multiple environment variables can be specified in the dialog.

  6. Click Submit.

Troubleshooting

Jenkins plugin cannot start uploading data ("Could not create session for closed launch" message in log)

In some cases you may encounter a situation where Jenkins plugin cannot upload data to Allure TestOps and writes a "Could not create session for closed launch" message into the log.

This is related to the fact that Jenkins pipelines do not have a unique identifier other than their name. Hence, if you delete a pipeline and create a new one with the same name, you would not be able to upload data to Allure TestOps because that name has already been used.

To fix that, simply rename the pipeline to something that has not been used yet.

Jenkins plugin cannot start uploading data (java.lang.UnsupportedClassVersionError)

Problem

Errors occurred that caused problems with uploading test results:

java.lang.UnsupportedClassVersionError: io/qameta/allure/jenkins/callable/UploadResultsCallable has been compiled by a more recent version of
the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0

The errors are followed by:

Caused: hudson.remoting.ProxyException: io.qameta.allure.jenkins.AllurePluginException: Could not stop uploader

Reason

It means you are using the Allure TestOps plugin incompatible with the current Jenkins CI Java version (Jenkins Java is newer than the version required to start the Allure TestOps plugin).

Solution

You need to upgrade the Allure TestOps plugin for Jenkins to the release compatible with Java 17.

Please also check your Jenkins CI version.

© Qameta Software Inc. All rights reserved.