TestOps logo
Try TestOps

Integration with GitHub

This page describes how to configure Allure TestOps integration with GitHub in a project that uses GitHub to run tests. This integration supports both the main GitHub server (github.com) and GitHub Enterprise Server installations.

Integration with GitHub allows you to set up the following relations:

  • One job in Allure TestOps corresponds to one GitHub workflow.
  • One launch in Allure TestOps corresponds to one GitHub workflow run.

A new workflow run can be triggered by either Allure TestOps or GitHub itself, with both parties displaying its status in their web interfaces.

To add Allure TestOps support to your GitHub project:

  1. Enable sending data from GitHub.
  2. Enable triggering GitHub workflows.
  3. Parametrize workflows (if necessary).

1. Enable sending data from GitHub

For GitHub to send workflow statuses and test results to Allure TestOps, you need to generate an API token in Allure TestOps, add the token to GitHub, and modify the workflow itself.

Then, run and check the workflow to make sure that everything works so far.

1.1. Create a token in Allure TestOps

  1. In Allure TestOps, click your avatar and go to API Tokens.

  2. Click + Token.

  3. Enter a name for the token (e.g., "Token for GitHub") and click Create.

    Allure TestOps will generate the token and display it in a modal window.

  4. Click the Copy icon to copy the token into clipboard. You will need this token in the next step.

1.2. Specify the token in GitHub

  1. In GitHub, open the project and go to its Settings.

  2. In the menu on the left, click Secrets and variables → Actions.

  3. Click New repository secret.

  4. Fill in the fields:

    • Name — ALLURE_TOKEN.
    • Secret — the API token that you got in step 1.1.

  5. Click Add secret.

1.3. Modify the workflows

To make changes to the workflow, edit the YAML file in the .github/workflows directory of your GitHub repository. You can do it either locally or using the GitHub web-based editor.

For each workflow that runs tests, do the following:

  1. Declare inputs for the workflow_dispatch event:

    • ALLURE_JOB_RUN_ID
    • ALLURE_USERNAME

    These variables are required for triggering workflows from Allure TestOps side. Without them you will get error 422.

    You will also get error 422 if the job parameters specified in Allure TestOps (on the left side of the form) do not match the inputs for workflow_dispatch.

  2. Add or extend the env section of your workflow. It must include the following:

    • ALLURE_ENDPOINT — URL of your Allure TestOps server (e.g., https://testops.example.com/).
    • ALLURE_PROJECT_ID — ID of your Allure TestOps project (e.g., 123). ID is displayed before the project name in the list of projects in the Allure TestOps interface. You can also find the project ID in the address bar of your browser when you are working with project in Allure TestOps.
    • ALLURE_TOKEN — ${{ secrets.ALLURE_TOKEN }}.
    • ALLURE_JOB_RUN_ID — ${{ github.event.inputs.ALLURE_JOB_RUN_ID }}.
    • ALLURE_RESULTS — path to the directory that contains test results (e.g., build/allure-results). If there are multiple test result directories in your project, you can separate them with commas or use a wildcard pattern (e.g., modules/*/build/allure-results).

  3. Add a step that will download allurectl before the step that runs the tests.

    We recommend using the allure-framework/setup-allurectl@v1 GitHub Action for this.

    Alternatively, you can download and configure allurectl without using the GitHub Action. You can find an example in the Examples section below.

  4. Wrap the command that runs the tests in an allurectl watch call.

    If your Allure TestOps server uses a self-signed SSL certificate, use allurectl --insecure watch instead.

Example

Assume we have a Java project with a workflow file that looks like this:

name: Run tests

on:
  push:

jobs:
  all-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up JDK
        uses: actions/setup-java@v3
        with:
          distribution: oracle
          java-version: 17

      - name: Build with Gradle
        run: ./gradlew clean test

To add Allure TestOps integration, we need to modify the file to look similar to this:

name: Run tests

on:
  push:
  workflow_dispatch:
    inputs:
      ALLURE_JOB_RUN_ID:
        description: ALLURE_JOB_RUN_ID service parameter. Leave blank.
      ALLURE_USERNAME:
        description: ALLURE_USERNAME service parameter. Leave blank.

env:
  ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
  ALLURE_JOB_RUN_ID: ${{ github.event.inputs.ALLURE_JOB_RUN_ID }}
  ALLURE_ENDPOINT: https://testops.example.com
  ALLURE_PROJECT_ID: 1
  ALLURE_RESULTS: build/allure-results

jobs:
  all-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up JDK
        uses: actions/setup-java@v3
        with:
          distribution: oracle
          java-version: 17

      - name: Install allurectl
        uses: allure-framework/setup-allurectl@v1

      - name: Build with Gradle
        run: allurectl watch -- ./gradlew clean test

The allurectl GitHub Action takes the following parameters:

  • allure-endpoint (required) — URL address of your Allure TestOps instance.
  • allure-token (required) — API token that will be used for authentication when uploading test results.
  • allure-project-id (required) — ID of your project in Allure TestOps.
  • allurectl-version (optional) — version of allurectl that should be used. See Releases for the list of available releases.
  • github-token (optional) — alternative GitHub token.

1.4. Run and check the workflow

  1. In GitHub, open your project and go to the workflow run triggered by the latest commit.

  2. If the run is not yet finished, wait until it finishes.

  3. In the run details, click the job that runs tests.

    Close to the end of its log, there should be a link to the test report in Allure TestOps. Make sure that it is present and works.

    The link to the launch is in the workflow run log.

  4. In the test report in Allure TestOps, open any test result.

    At the bottom of the page, there should be a link back to the GitHub workflow run. Make sure that it is present and works.

    The link to the GitHub workflow run is at the bottom of the right pane.

2. Enable triggering GitHub workflows

On the Allure TestOps side, the integration with GitHub needs to be configured on two levels.

First, the administrator needs to specify the URL of the GitHub server.

Then, a project owner needs to create an authentication token in GitHub and add it to Allure TestOps.

2.1. Specify GitHub server in Allure TestOps

  1. Log in to Allure TestOps using an administrator account.

  2. Go to Administration → Integrations.

  3. Click + Add integration in the top right corner of the page.

  4. In the dialog that appears, select GitHub.

  5. Fill in the fields:

    • Name — a name to help you recognize the GitHub server (e.g., “GitHub production”).
    • Endpoint — the base URL of GitHub. For github.com, use “https://github.com”. For GitHub Enterprise Server, use the URL of your GitHub instance.
    • Endpoint for API calls — the URL of GitHub API. For github.com, use “https://api.github.com”. For GitHub Enterprise Server, use “⟨URL⟩/api/v3”, where ⟨URL⟩ is the URL of your GitHub instance.

  6. If you are using GitHub Enterprise Server with a self-signed SSL certificate, check the Disable certificate validation box.

  7. Click Add integration.

2.2. Create a token in GitHub

To be able to trigger workflows, Allure TestOps needs to have a personal access token generated by the GitHub server. You can choose the type of token that GitHub will generate: fine-grained or classic (see Managing your personal access tokens in the GitHub documentation).

Instructions for generating a token vary slightly depending on the type.

  1. In GitHub, click your avatar and go to Settings.

  2. In the menu on the left, click Developer settings.

  3. Go to Personal access tokens → Fine-grained tokens.

  4. Click Generate new token.

  5. Fill in the fields:

    • Token name — a name to help you recognize the token (e.g., “Token for Allure TestOps”).
    • Expiration — specify how long the token should be valid. After this date, the integration will stop working, and you will need to create a new token to continue using the integration.

  6. Under the Repository access section, click Only select repositories. In the drop-down list that appears, select the repositories you want to use.

  7. Under the Permissions section, click Repository permissions. In the permission list that appears, find Actions and select Read and write next to it.

    If you plan to add links to GitHub issues, additionally find Issues and select Read and write next to it.

  8. Click Generate token.

    The page will reload, and the new token will become temporarily visible. Click the Copy icon next to it.

    You will need this token on the next step.

  1. In GitHub, click your avatar and go to Settings.

  2. In the menu on the left, click Developer settings.

  3. Go to Personal access tokens → Tokens (classic).

  4. Click Generate new token → Generate new token (classic).

  5. Fill in the fields:

    • Note — a name to help you recognize the token (e.g., “Token for Allure TestOps”).
    • Expiration — specify how long the token should be valid. After this date, the integration will stop working, and you will need to create a new token to continue using the integration.

  6. Under the Select scopes section, check workflow.

    If you plan to add links to GitHub issues, additionally check repo.

  7. Click Generate token.

    The page will reload, and the new token will become temporarily visible. Click the Copy icon next to it.

    You will need this token on the next step.

2.3. Add the token to an Allure TestOps project

  1. In Allure TestOps, open your project.
  2. Go to Settings → Integrations.
  3. Under Available integrations, find the GitHub integration and click Add integration next to it.
  4. Under Secret, enter the GitHub access token you created in step 2.2.
  5. Click Test connection. If the token is correct, a "Connection established" message will appear within a few seconds.
  6. Click Add integration to save the settings.

3. Parametrize workflows

GitHub uses the inputs section of the YAML file for passing parameters to workflow context. In order to parametrize the workflow, you need to add a variable to that section and then to the env section to make it available as an environment variable.

Allure TestOps integrates this feature with its Environment concept, which lets you both set parameters for new jobs, and see parameters set in workflows that were started from CI side.

Regardless of the tool you are using to upload test results to Allure TestOps (allurectl or a CI plugin), the context of the GitHub workflow will be uploaded as well and all environment variables will be linked to the corresponding test results.

3.1. Declare inputs in the GitHub workflow

To make changes to the workflow, edit the YAML file in the .github/workflows directory of your GitHub repository. You can do it either locally or using the GitHub web-based editor.

Add workflow_dispatch inputs, their descriptions, and default values to the inputs section. For example:

name: Run tests

on:
  workflow_dispatch:
    inputs:
      ALLURE_JOB_RUN_ID:
        description: ALLURE_JOB_RUN_ID service parameter. Leave blank.
      ALLURE_USERNAME:
        description: ALLURE_USERNAME service parameter. Leave blank.
      TESTS_BROWSER:
        description: Browser to use for tests
        default: chrome
      PRODUCT_VERSION:
        description: Product version
        default: '1.23'
...

Then, specify them in the env section to make them available as environment variables:

env:
  PRODUCT_VERSION: ${{ github.event.inputs.PRODUCT_VERSION }}
  TESTS_BROWSER: ${{ github.event.inputs.TESTS_BROWSER }}

These environment variables will be sent along with the others to Allure TestOps.

3.2. Add global environment variables

Before using environment data from a GitHub workflow (or from any kind of pipeline in general), you need to create a global environment variable in Allure TestOps which will store that data.

  1. Log in to Allure TestOps using an administrator account.

  2. Go to Administration → Environment.

  3. For each variable that you want to add:

    1. Click + Create.
    2. Enter a name for the variable.
    3. Click Submit.

    Environment variable names are added in the Administration section.

3.3. Map workflow environment variables to global environment variables

Once you've declared environment variables in your workflow and added global variables in Allure TestOps, you can configure your Allure TestOps project to start receiving this data:

  1. In Allure TestOps, open your project.

  2. Go to Settings → Environment.

  3. For each variable that you want to use:

    1. Click + Create if the variable is not in the list. If the variable already exists, click the Edit icon next to its name.
    2. In the Mapping key field, specify the name of the environment variable.
    3. In the Environment variable field, select the global name from step 3.2.
    4. Click Submit.

    Mapping environment variables is configured in the project settings.

3.4. Add parameters to the job

  1. In Allure TestOps, open your project.

  2. Go to Jobs.

  3. Find the job you want to parametrize. Click ⋯ next to the job name and select Configure.

    Job settings dialog will appear, which contains the Parameters section.

  4. For each variable that you want to add, click Add and fill in the fields:

    • Name — name of the environment variable (same as Mapping key from step 3.3).
    • Value — default value that should be used if not specified for a launch.
    • Environment Variable — environment variable from step 3.2.

    Multiple environment variables can be specified in the dialog.

  5. Click Submit.

Note that it is recommended to set the same default values in Allure TestOps and GitHub. This way, the workflow run will get the same environment data regardless of what triggered it.

3.5. Launch a workflow with specific environment variables

In the previous steps, we configured an Allure TestOps job, adding all the parameters we want to pass to our GitHub workflow when launching it from Allure TestOps. These parameters will be listed next to the job name in the list.

The parameters are listed next to the job name.

Each listed parameter contains the following:

  • name of the environment variable that will be available in the CI workflow (TESTS_BROWSER),
  • name of the global environment variable that will be displayed in Allure TestOps interface (Browser),
  • default value for the environment variable (chrome).

To launch a parametrized workflow, use one of the following methods:

You should see the following dialog:

The “Add” button is at the bottom of the dialog.

To specify an environment variable, click the + Add button under Environment and enter the required values.

The parameter values can be entered or selected from the list.

If you need to launch your workflow multiple times using different environment data, you can click the bottom + Add button again to create an additional set of environment variables. Allure TestOps will trigger the GitHub workflow multiple times, once for each set of environment variables.

Please note that when you rerun a test, it is always executed using the same environment data as the original run.

Examples

Below you can find a simple example that you can use as a basis for your GitHub workflow. It uses pytest as a testing framework and contains all the required settings to upload the test results to Allure TestOps and to trigger workflows from the Allure TestOps interface.

As was previously mentioned, there are two ways you can add allurectl to your workflow:

  • Using GitHub Action:

    1. Action will download allurectl.
    2. Action will add parameters based on the workflow context or the context provided by Allure TestOps.
    3. Action will set up the parameters required for uploading test results.

  • Using a script inside your workflow (this option can be used if GitHub Action is unavailable due to restrictions):

    1. Script will download allurectl.
    2. Script will add parameters based on the workflow context.
name: run-and-upload-to-testops

on:
  workflow_dispatch:
    inputs:
      GITHUB_TESTS_ENDPOINT:
        description: "System under test"
        required: true
        default: https://system.under.test
      GITHUB_TESTS_BROWSER:
        description: "Browser to be used in tests"
        required: true
        default: chrome
      ALLURE_JOB_RUN_ID:
        description: "ALLURE_JOB_RUN_ID is service parameter required for triggering workflows from Allure TestOps. Leave blank."
        required: false
      ALLURE_USERNAME:
        description: "ALLURE_USERNAME service parameter. Leave blank"
        required: false


env:
  ALLURE_RESULTS: "allure-results"
  ALLURE_JOB_RUN_ID: ${{ github.event.inputs.ALLURE_JOB_RUN_ID }}

jobs:
  all-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - name: Install dependencies for pytest and allure framework integration
        run: |
          python -m pip install --upgrade pip
          pip install allure-pytest pytest
      - name: install and configure allurectl using GH Action
        uses: allure-framework/setup-allurectl@v1
        with:
          allure-endpoint: https://testops.example.com
          allure-token: ${{ secrets.ALLURE_TOKEN }}
          allure-project-id: 9999
      - name: Run pytest tests
        run: |
          allurectl watch -- pytest ./test --alluredir=${ALLURE_RESULTS} --capture=no || true
        env:
          GITHUB_TESTS_ENDPOINT: ${{ github.event.inputs.GITHUB_TESTS_ENDPOINT }}
          GITHUB_TESTS_BROWSER: ${{ github.event.inputs.GITHUB_TESTS_BROWSER }}
          GITHUB_TESTS_BRANCH: ${{ github.ref_name }}

name: pytest tests with allure framework

on:
  workflow_dispatch:
    inputs:
      GITHUB_TESTS_ENDPOINT:
        description: "System under test"
        required: true
        default: https://system.under.test
      GITHUB_TESTS_BROWSER:
        description: "Browser to be used in tests"
        required: true
        default: chrome
      ALLURE_JOB_RUN_ID:
        description: "Inner parameter for Allure TestOps"
        required: false
      ALLURE_USERNAME:
        description: "ALLURE_USERNAME service parameter. Leave blank"
        required: false

env:
  ALLURE_ENDPOINT: ${{ secrets.ALLURE_ENDPOINT }}
  ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
  ALLURE_PROJECT_ID: ${{ secrets.ALLURE_PROJECT_ID }}
  ALLURE_TESTPLAN_PATH: "./testplan.json"
  ALLURE_RESULTS: "allure-results"
  ALLURE_JOB_RUN_ID: ${{ github.event.inputs.ALLURE_JOB_RUN_ID }}
  GH_BRANCH: ${{ github.ref_name }}
  ALLURE_LAUNCH_TAGS: "github, demo, pytest, $GH_BRANCH"

jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
    - uses: actions/github-script@v4
      id: allure-job-uid
      with:
        result-encoding: string
        script: |
          const result = await github.actions.getWorkflowRun({
            owner: context.repo.owner,
            repo: context.repo.repo,
            run_id: context.runId,
          });
          return `${context.repo.owner}/${context.repo.repo}/actions/workflows/${result.data.workflow_id}`
    - name: Download and make allurectl executable
      run: |
        wget https://github.com/allure-framework/allurectl/releases/download/2.15.1/allurectl_linux_amd64 -O ./allurectl
        chmod +x ./allurectl
    - name: Install dependencies for pytest and allure framework integration
      run: |
        python -m pip install --upgrade pip
        pip install allure-pytest pytest
    - name: Test with pytest and gen allure results
      shell: bash
      working-directory: ${{ github.action_path }}
      env:
        ALLURE_JOB_UID: ${{steps.allure-job-uid.outputs.result}}
        GITHUB_TESTS_ENDPOINT: ${{ github.event.inputs.GITHUB_TESTS_ENDPOINT }}
        GITHUB_TESTS_BROWSER: ${{ github.event.inputs.GITHUB_TESTS_BROWSER }}
      run: |
        ./allurectl watch -- pytest --alluredir=${ALLURE_RESULTS}

Step id: allure-job-uid is required if you want to trigger the workflow from Allure TestOps.

Troubleshooting

Error 422 when triggering GitHub workflow from Allure TestOps interface

General cause

Usually, error 422 occurs when you are trying to launch a workflow using workflow_dispatch and the parameters defined in inputs of workflow_dispatch do not match the parameters sent by Allure TestOps.

Expected configuration

As mentioned earlier, there are at least two parameters (with the required attribute set to false) that must be included as inputs in your workflow_dispatch section in order for Allure TestOps to have the ability to trigger the workflow:

  • ALLURE_JOB_RUN_ID,
  • ALLURE_USERNAME.

The values for these parameters will be set by Allure TestOps automatically.

All additional parameter values (e.g., browser, system under test, etc.) must be specified in the Job configuration dialog when launching the workflow using the Allure TestOps interface.

Example

name: Integration of Allure TestOps with GitHub 

on:
  workflow_dispatch:
    inputs:
      ALLURE_JOB_RUN_ID:
        description: "ALLURE_JOB_RUN_ID service parameter. Leave blank."
        required: false
      ALLURE_USERNAME:
        description: "ALLURE_USERNAME service parameter. Leave blank"
        required: false
      TEST_BROWSER:
        description: "Browser for tests"
        required: true
        default: chrome
      TEST_ENDPOINT:
        description: "System under test"
        required: false
        default: https://staging.system.under.test

For the above configuration, you need to set the following parameters in Allure TestOps:

  1. TEST_BROWSER

    Must be set because of required: true.

  2. TEST_ENDPOINT

    Can be set or omitted because of required: false.

  3. Branch

    Must be set. Will be linked to the Branch environment variable (both starting with capital B).

    This is a special case variable that is not explicitly listed in inputs but must be set because it is required by the API. Its value will be used for the ref parameter.

Examples of GitHub behavior when trying to run a job
Will work correctly.

All parameters are in sync with inputs
Will work correctly.

TEST_BROWSER parameter is set, as required by inputs.
TEST_ENDPOINT is not set, but it is not required.
Will not work and will cause error 422.

RELEASE_ID parameter is set but does not exist in inputs.
Will not work and will cause error 422.

TEST_BROWSER parameter is not set, but is required by inputs.

Solution

The only possible solution is to synchronize the configuration parameters of the job created in Allure TestOps with your inputs in workflow_dispatch.

© Qameta Software Inc. All rights reserved.