Integration with GitHub

The integration between Allure TestOps and GitHub allows you to set up the following relations:

• One job in Allure TestOps corresponds to one GitHub workflow.
• One launch in Allure TestOps corresponds to one GitHub workflow run.

A new workflow run can be triggered by either Allure TestOps or GitHub itself, with both parties displaying its status in their web interfaces.

The integration supports both the main GitHub server (github.com) and GitHub Enterprise Server installations.

For the integration to work properly, you will need to modify your workflows so that they use the setup-allurectl Action. This Action will set up the allurectl utility to communicate with Allure TestOps. Before running the tests, the utility receives the test plan, i.e. the selection of tests that need to run. Then, while the tests are running, the utility regularly scans for new files in the test results directory (e.g., “build/allure-results”, see Allure Report → How it works) and uploads them to the Allure TestOps server. This way, Allure TestOps receives the test results as soon as possible and is able to show a launch's partial results even before the job is finished.

To add Allure TestOps support in your GitHub project:

1. Enable sending data from GitHub.
2. Enable triggering GitHub workflows.
3. Parametrize workflows (if necessary).

1. Enable sending data from GitHub

For GitHub to send workflow statuses and test results to Allure TestOps, you need to generate an authentication token in Allure TestOps, add the token to GitHub, and modify the workflow itself.

Then, run and check the workflow to make sure that everything works so far.

1.1. Create a token in Allure TestOps

1. In Allure TestOps, click your avatar and go to API Tokens.

2. Click + Token.

3. Enter a name for the token (e.g., "Token for GitHub") and click Create.

   Allure TestOps will generate the token and display it in a modal window.

4. Click the Copy icon to copy the token into clipboard. You will need this token in the next step.

1.2. Specify the token in GitHub

1. In GitHub, open the project and go to its Settings.

2. In the menu on the left, click Secrets and variables → Actions.

3. Click New repository secret.

4. Fill in the fields:

   • Key — “ALLURE_TOKEN”.
   • Value — the API token that you got in step 1.1.

5. Click Add secret.

1.3. Modify the workflows

To make changes to a workflow, edit its YAML file in the .github/workflows directory of your GitHub repository. You can do it either locally or in the GitHub's web-based text editor.

For each workflow that runs tests, do the following:

1. Declare workflow inputs for the workflow_dispatch event:

   • ALLURE_JOB_RUN_ID
   • ALLURE_USERNAME

   It's mandatory to add these variables. If the variables won't be added, triggering of workflows from Allure TestOps side won't work and you will get 422 error.

   You will also get 422 error if the job parameters specified in Allure TestOps (on the left side of the form) do not match the inputs for workflow_dispatch.

2. Add or extend the workflow's env block. It must include:

   • ALLURE_ENDPOINT — the URL of the Allure TestOps server (e.g., “https://demo.testops.cloud/”).
   • ALLURE_PROJECT_ID — the ID of the Allure TestOps project, e.g., “1”. Id stays before the project name in Allure TestOps UI, and the id is present in the address field when you are working in Allure TestOps.
   • ALLURE_TOKEN — ${{ secrets.ALLURE_TOKEN }}.
   • ALLURE_JOB_RUN_ID — ${{ github.event.inputs.ALLURE_JOB_RUN_ID }}.
   • ALLURE_RESULTS — the path to the test results directory (e.g., “build/allure-results”). If there are multiple test result directories in your project, you can separate them with commas or use a wildcard pattern, e.g., “modules/*/build/allure-results”.

3. Before running tests, add two new steps:

   1. A step that downloads the allurectl utility. It is recommended to use the allure-framework/setup-allurectl@v1 Action for this.
   2. There is another option to download and setup allurectl without this Action. We'll provide both examples below.

4. Wrap the command that runs the tests into the allurectl watch command.

   If your Allure TestOps server uses a self-signed SSL certificate, use allurectl --insecure watch instead.

Example

Assume we have a Java project with a workflow file like this:

name: Run tests

on:
  push:

jobs:
  all-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up JDK
        uses: actions/setup-java@v3
        with:
          distribution: oracle
          java-version: 17

      - name: Build with Gradle
        run: ./gradlew clean test

With Allure TestOps integration, the file will look something like this:

name: Run tests

on:
  push:
  workflow_dispatch:
    inputs:
      ALLURE_JOB_RUN_ID:
        description: ALLURE_JOB_RUN_ID service parameter. Leave blank.
      ALLURE_USERNAME:
        description: ALLURE_USERNAME service parameter. Leave blank.

env:
  ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
  ALLURE_JOB_RUN_ID: ${{ github.event.inputs.ALLURE_JOB_RUN_ID }}
  ALLURE_ENDPOINT: https://demo.testops.cloud/
  ALLURE_PROJECT_ID: 1
  ALLURE_RESULTS: build/allure-results

jobs:
  all-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up JDK
        uses: actions/setup-java@v3
        with:
          distribution: oracle
          java-version: 17

      - name: Install allurectl
        uses: allure-framework/setup-allurectl@v1

      - name: Build with Gradle
        run: allurectl watch -- ./gradlew clean test

allurectl setup action takes the following parameters:

• allure-endpoint Mandatory parameter, setting the URL address of Allure TestOps instance to upload the test results to.
• allure-token Mandatory parameter used for the authentication of allurectl at Allure TestOps server defined by allure-endpoint parameter.
• allure-project-id Mandatory parameter to define the project (numeric ID is used) where launch with test results has to be created.
• allurectl-version Optional parameter allowing you to select specific version of allurectl to be used. Information on the available releases can be found here.
• github-token Optional parameter to provide an alternative GitHub token.

1.4. Run and check the workflow

1. In GitHub, open the project and go to the workflow run triggered by the latest commit.

2. If the run is not yet finished, wait until it finishes.

3. In the run details, click the job that runs tests.

   Close to the end of its log, there should be a link to the test report at Allure TestOps. Make sure that it is present and works.

   

4. In the test report at Allure TestOps, open a single test's results.

   At the bottom of the page, there should be a link back to the GitHub workflow run. Make sure that it is present and works.

   

2. Enable triggering GitHub workflows

On the Allure TestOps side, the integration with GitHub needs to be configured on two levels.

First, the administrator needs to specify the URL of GitHub server.

Then, any project owner needs to create an authentication token in GitHub and add it to Allure TestOps.

2.1. Specify GitHub server in Allure TestOps

1. Log in to Allure TestOps using an administrator account.

2. Go to Administration → Integrations.

3. Click + Add integration in the top right corner of the page.

4. In the dialog that appears, select GitHub.

5. Fill in the fields:

   • Name — a name to help you recognize the GitHub server (e.g., “GitHub production”).
   • Endpoint — the base URL of GitHub. For github.com, use “https://github.com”. For GitHub Enterprise Server, use the URL of your GitHub instance.
   • Endpoint for API calls — the URL of GitHub API. For github.com, use “https://api.github.com”. For GitHub Enterprise Server, use “⟨URL⟩/api/v3”, where ⟨URL⟩ is the URL of your GitHub instance.

6. If you are using GitHub Enterprise Server with a self-signed SSL certificate, check the Disable certificate validation box.

7. Click Add integration.

2.2. Create a token in GitHub

To be able to trigger workflows, Allure TestOps needs to have a personal access token generated by the GitHub server. You can choose the type of token that GitHub will generate: fine-grained or classic (see Managing your personal access tokens in the GitHub documentation).

Instructions for generating a token vary slightly depending on the type.

2.3. Add the token to an Allure TestOps project

1. In Allure TestOps, open your project.
2. Go to Settings → Integrations.
3. Under Available integrations, find the GitHub integration and click Add integration next to it.
4. Under Secret, enter the GitHub access token you created in step 2.2.
5. Click Test connection. If the token is correct, a "Connection established" message will appear within a few seconds.
6. Click Add integration to save the settings.

3. Parametrize workflows

GitHub uses inputs for passing parameters to workflows' context, i.e. you need to create an input variable which value then will be assigned to an environment variable in the context of your workflow execution and then this value can be used in the tests.

Allure TestOps integrates this feature with its Environment concept, which lets you both set parameters for new jobs, and see parameters set in workflows that were started from CI side.

Despite the tool (allurectl or a CI plugin) used to upload the test results to Allure TestOps, whole pipeline's context (in case of GitHub – the context of a workflow) is transferred to Allure TestOps, and can be used to link environment data to the uploaded test results.

3.1. Declare inputs in the GitHub workflow

To make changes to a workflow, edit its YAML file in the .github/workflows directory of your GitHub repository. You can do it either locally or in the GitHub's web-based text editor.

Add workflow dispatch inputs, their descriptions, and default values into the inputs block. For example:

name: Run tests

on:
  workflow_dispatch:
    inputs:
      ALLURE_JOB_RUN_ID:
        description: ALLURE_JOB_RUN_ID service parameter. Leave blank.
      ALLURE_USERNAME:
        description: ALLURE_USERNAME service parameter. Leave blank.
      TESTS_BROWSER:
        description: Browser to use for tests
        default: chrome
      PRODUCT_VERSION:
        description: Product version
        default: '1.23'
...

You can use these inputs later in your workflow to pass the information to your tests via environment variables.

        env:
          PRODUCT_VERSION: ${{ github.event.inputs.PRODUCT_VERSION }}
          TESTS_BROWSER: ${{ github.event.inputs.TESTS_BROWSER }}

These environment variables, along with others, will be sent by allurectl to Allure TestOps within the build context.

3.2. Add global environment variables names

Before using the environment variable data from a GitHub workflow, or generally from any kind of pipeline, you need to create a global environment variable which will store such a data.

1. Log into Allure TestOps using an administrator account.

2. Go to Administration → Environment.

3. For each variable that you want to add:

   1. Click + Create.

   2. Enter the variable name.

   3. Click Submit.

   

3.3. Map workflow environment variables to global environment variables

After you have an environment variable declared in your workflow (pipeline), and created global environment variable in Allure TestOps, you can start configuring your project environment variables to extract the information you need from workflow's context.

1. In Allure TestOps, open your project.

2. Go to Settings → Environment.

3. For each variable that you want to use:

   1. Click + Create if the variable is not in the list. If the variable already exists, click the Edit icon next to its name.

   2. In the Mapping key field, specify the name of the environment variable.

   3. In the Environment variable field, select the global name from step 3.2.

   4. Click Submit.

   

3.4. Add parameters to the job

1. In Allure TestOps, open your project.

2. Go to Jobs.

3. Find the job which you want to parametrize. Click ⋯ next to the job name, then select Configure.

   The job settings dialog will appear, containing the Parameters section.

4. For each variable that you want to add, click Add and fill in the fields:

   • Name — the name of the environment variable (same as the Mapping key from step 3.3).
   • Value — the default value that should be used unless overridden for a specific launch.
   • Environment Variable — the environment variable from step 3.2.

   

5. Click Submit.

Note that it is recommended to set the same default values both in Allure TestOps and GitHub. This means that a workflow run will get the same environment regardless of what triggered it.

3.5. Executing a workflow with certain environment data

In the previous steps we configured a job on Allure TestOps side, which now has all the parameters we want to pass to our GitHub workflow.

Configured job parameters are the ones, you will be able to pass to your pipeline when triggering CI pipelines from Allure TestOps UI.

Parameter rendered as the first one in the list, is the name of an environment variable that we are going to set on CI side (e.g. TEST_BROWSER). The parameter in the parenthesis in the name of a global environment variable which will provide the suggestions for the values (e.g. Browser). And the value (in our example it's "chrome") is the default value we are going to push to a CI system if no alternative value.

To execute tests on certain environment you need to invoke one of the following actions:

• bulk run of test cases;
• execution of a test plan;
• execution of the entire job (this section).

Please note, reruns of the tests are always executed on the same environment as it was set during the initial execution.

When you execute any of the said actions, you will see the following form for the creation of a launch.

You need to click the button saying Add in the Environment section.

Application will add the job parameters to the form and pre-fill these with the job parameters default values.

Then you can alter the values you want to pass to the CI pipeline.

If you need to execute your pipeline several times on different sets of the parameters, you need to create a new set of the environment variables and Allure TestOps will request CI to run several pipelines with different sets of parameters.

Examples

Below is a very simple workflow, you can use as a start for your GitHub workflow to try the execution of tests and uploading the data to Allure TestOps. It uses pytest as an example, and it contains all the elements required to upload the test results to Allure TestOps and allow the triggering of workflows from Allure TestOps side.

As we mentioned before, there are two ways to add allurectl to the workflow.

• using GitHub Action:
  1. Action will download allurectl.
  2. Action will generate all the required parameters based on workflow context or context provided from Allure TestOps.
  3. Action will provide configuration parameters required for the test results upload.
• using script inside your workflow (this option is used when GitHub cannot be used due to certain restrictions):
  1. Script will download allurectl.
  2. Script will generate the required parameters based on workflow context.

name: run-and-upload-to-allure-testops

on:
  workflow_dispatch:
    inputs:
      GITHUB_TESTS_ENDPOINT:
        description: "System under test"
        required: true
        default: https://system.under.test
      GITHUB_TESTS_BROWSER:
        description: "Browser to be used in tests"
        required: true
        default: chrome
      ALLURE_JOB_RUN_ID:
        description: "ALLURE_JOB_RUN_ID is service parameter required for triggering workflows from Allure TestOps. Leave blank."
        required: false
      ALLURE_USERNAME:
        description: "ALLURE_USERNAME service parameter. Leave blank"
        required: false


env:
  ALLURE_RESULTS: "allure-results"
  ALLURE_JOB_RUN_ID: ${{ github.event.inputs.ALLURE_JOB_RUN_ID }}

jobs:
  all-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - name: Install dependencies for pytest and allure framework integration
        run: |
          python -m pip install --upgrade pip
          pip install allure-pytest pytest
      - name: install and configure allurectl using GH Action
        uses: allure-framework/setup-allurectl@v1
        with:
          allure-endpoint: https://demo.testops.cloud
          allure-token: ${{ secrets.ALLURE_TOKEN }}
          allure-project-id: 9999
      - name: Run pytest tests
        run: |
          allurectl watch -- pytest ./test --alluredir=${ALLURE_RESULTS} --capture=no || true
        env:
          GITHUB_TESTS_ENDPOINT: ${{ github.event.inputs.GITHUB_TESTS_ENDPOINT }}
          GITHUB_TESTS_BROWSER: ${{ github.event.inputs.GITHUB_TESTS_BROWSER }}
          GITHUB_TESTS_BRANCH: ${{ github.ref_name }}

name: pytest tests with allure framework

on:
  workflow_dispatch:
    inputs:
      GITHUB_TESTS_ENDPOINT:
        description: "System under test"
        required: true
        default: https://system.under.test
      GITHUB_TESTS_BROWSER:
        description: "Browser to be used in tests"
        required: true
        default: chrome
      ALLURE_JOB_RUN_ID:
        description: "Inner parameter for Allure TestOps"
        required: false
      ALLURE_USERNAME:
        description: "ALLURE_USERNAME service parameter. Leave blank"
        required: false

env:
  ALLURE_ENDPOINT: ${{ secrets.ALLURE_ENDPOINT }}
  ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
  ALLURE_PROJECT_ID: ${{ secrets.ALLURE_PROJECT_ID }}
  ALLURE_TESTPLAN_PATH: "./testplan.json"
  ALLURE_RESULTS: "allure-results"
  ALLURE_JOB_RUN_ID: ${{ github.event.inputs.ALLURE_JOB_RUN_ID }}
  GH_BRANCH: ${{ github.ref_name }}
  ALLURE_LAUNCH_TAGS: "github, demo, pytest, $GH_BRANCH"

jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
    - uses: actions/github-script@v4
      id: allure-job-uid
      with:
        result-encoding: string
        script: |
          const result = await github.actions.getWorkflowRun({
            owner: context.repo.owner,
            repo: context.repo.repo,
            run_id: context.runId,
          });
          return `${context.repo.owner}/${context.repo.repo}/actions/workflows/${result.data.workflow_id}`
    - name: Download and make allurectl executable
      run: |
        wget https://github.com/allure-framework/allurectl/releases/download/2.15.1/allurectl_linux_amd64 -O ./allurectl
        chmod +x ./allurectl
    - name: Install dependencies for pytest and allure framework integration
      run: |
        python -m pip install --upgrade pip
        pip install allure-pytest pytest
    - name: Test with pytest and gen allure results
      shell: bash
      working-directory: ${{ github.action_path }}
      env:
        ALLURE_JOB_UID: ${{steps.allure-job-uid.outputs.result}}
        GITHUB_TESTS_ENDPOINT: ${{ github.event.inputs.GITHUB_TESTS_ENDPOINT }}
        GITHUB_TESTS_BROWSER: ${{ github.event.inputs.GITHUB_TESTS_BROWSER }}
      run: |
        ./allurectl watch -- pytest --alluredir=${ALLURE_RESULTS}

Troubleshooting

Error 422 when triggering GitHub workflow via Allure TestOps interface

Root cause

Generally, the error 422 is returned by GitHub when you are trying to start workflow using workflow_dispatch, and the parameters defined in the inputs of workflow_dispatch do not match those sent from Allure TestOps.

Expected configuration

As mentioned earlier in the instruction, there are at least two mandatory parameters (with the required attribute set to false) that must be included as inputs in the workflow_dispatch section for Allure TestOps to trigger a workflow:

• ALLURE_JOB_RUN_ID,
• ALLURE_USERNAME.

These parameters are set by Allure TestOps automatically without involving any user's actions. Both are necessary for successfully triggering workflows.

If your pipeline (workflow) logic requires additional mandatory data (e.g., browser name, system under test name), these parameters must be configured in Allure TestOps as job configuration parameters.

Example

name: Integration of Allure TestOps with GitHub 

on:
  workflow_dispatch:
    inputs:
      ALLURE_JOB_RUN_ID:
        description: "ALLURE_JOB_RUN_ID service parameter. Leave blank."
        required: false
      ALLURE_USERNAME:
        description: "ALLURE_USERNAME service parameter. Leave blank"
        required: false
      TEST_BROWSER:
        description: "Browser for tests"
        required: true
        default: chrome
      TEST_ENDPOINT:
        description: "System under test"
        required: false
        default: https://staging.system.under.test

For the described configuration of inputs for workflow_dispatch on the Allure TestOps side, you must have the following configuration parameters for the created job:

1. Branch:

   • Linked to the Branch environment variable (both starting with capital B).
   • This is a special environment variable not explicitly listed in the inputs, but its value is passed to ref parameter in the API call. Therefore, this variable is mandatory (cannot be omitted).

2. TEST_BROWSER:

   • Passes the browser name to workflow_dispatch.
   • Must be included in the job configuration parameters as it is defined in the workflow_dispatch as required: true.

3. TEST_ENDPOINT:

   • May be included as an optional job configuration parameter as it is defined in the workflow_dispatch as required: false.

Examples of GitHub behavior when you try executing the configured job

Will work correctly. All parameters are present in inputs
Will work correctly. The TEST_BROWSER parameter is present, which is mandatory for inputs in workflow_dispatch
Won't work, error 422. The RELEASE_ID parameter is not present in inputs of workflow_dispatch
Won't work, error 422. The TEST_BROWSER parameter is missing, which is mandatory for inputs in workflow_dispatch

Solution

The only possible solution is to synchronize the configuration parameters of the job created in Allure TestOps with the inputs in workflow_dispatch.