TestOps logo
Try TestOps

Integration with GitHub

The integration between Allure TestOps and GitHub allows you to set up the following relations:

  • one job in Allure TestOps corresponds to one GitHub workflow,
  • one launch of an Allure TestOps job corresponds to one GitHub workflow run.

A new workflow run can be triggered by either Allure TestOps or GitHub itself, with both parties displaying its status in their web interfaces.

The integration supports both the main GitHub server (github.com) and GitHub Enterprise Server installations.

For the integration to work properly, you will need to modify your workflows so that they use the setup-allurectl action. This action will set up the allurectl utility to communicate with Allure TestOps. Before running the tests, the utility receives the test plan, i.e. the selection of tests that need to run. Then, while the tests are running, the utility regularly scans for new files in the test results directory (e.g., “build/allure-results”, see Allure Report → How it works) and uploads them to the Allure TestOps server. This way, Allure TestOps receives the test results as soon as possible and is able to show a launch's partial results even before the job is finished.

To enable Allure TestOps support in your GitHub project:

  1. enable sending data from GitHub,
  2. enable triggering GitHub workflows,
  3. parametrize workflows (if necessary).

1. Enable sending data from GitHub

For GitHub to send workflow statuses and test results to Allure TestOps, you need to generate an authentication token in Allure TestOps, add the token to GitHub, and modify the workflow itself.

Then, run and check the workflow to make sure that everything works so far.

1.1. Create a token in Allure TestOps

  1. In Allure TestOps, click on your avatar and go to Your profile.

  2. Under the API tokens section, click + Token.

  3. Enter a Token name (e.g., “Token for GitHub”), then click Submit.

  4. In the dialog that appears, click the Copy icon to copy the token into clipboard.

    You will need this token on the next step.

1.2. Specify the token in GitHub

  1. In GitHub, open the project and go to its Settings.

  2. In the menu on the left, click Secrets and variables → Actions.

  3. Click New repository secret.

  4. Fill in the fields:

    • Key — “ALLURE_TOKEN”.
    • Value — the API token that you got on step 1.1.

  5. Click Add secret.

1.3. Modify the workflows

To make changes to a workflow, edit its YAML file in the .github/workflows directory of your GitHub repository. You can do it either locally or in the GitHub's web-based text editor.

For each workflow that runs tests, do the following:

  1. Declare workflow inputs for the workflow_dispatch event:

    • ALLURE_JOB_RUN_ID
    • ALLURE_USERNAME

    It's mandatory to add these variables. If the variables won't be added, triggering of workflows from Allure TestOps side won't work and you will get 422 error.

  2. Add or extend the workflow's env block. It must include:

  • ALLURE_ENDPOINT — the URL of the Allure TestOps server, e.g., “https://demo.testops.cloud/”.
  • ALLURE_PROJECT_ID — the ID of the Allure TestOps project, e.g., “1”. Id stays before the project name in Allure TestOps UI, and the id is present in the address field when you are working in Allure TestOps.
  • ALLURE_TOKEN — ${{ secrets.ALLURE_TOKEN }}.
  • ALLURE_JOB_RUN_ID — ${{ github.event.inputs.ALLURE_JOB_RUN_ID }}.
  • ALLURE_RESULTS — the path to the test results directory, e.g., “build/allure-results”. If there are multiple test result directories in your project, you can separate them with commas or use a wildcard pattern, e.g., “modules/*/build/allure-results”.

  1. Before running tests, add two new steps:

    1. A step that downloads the allurectl utility. It is recommended to use the allure-framework/setup-allurectl@v1 action for this.
    2. There is another option to download and setup allurectl without the said action. We'll provide both examples below.

  2. Wrap the command that runs the tests into the allurectl watch command.

    (If your Allure TestOps server uses a self-signed SSL certificate, use allurectl --insecure watch instead.)

Example

Assume we have a Java project with a workflow file like this:

name: Run tests

on:
  push:

jobs:
  all-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up JDK
        uses: actions/setup-java@v3
        with:
          distribution: oracle
          java-version: 17

      - name: Build with Gradle
        run: ./gradlew clean test

With Allure TestOps integration, the file will look something like this:

name: Run tests

on:
  push:
  workflow_dispatch:
    inputs:
      ALLURE_JOB_RUN_ID:
        description: ALLURE_JOB_RUN_ID service parameter. Leave blank.
      ALLURE_USERNAME:
        description: ALLURE_USERNAME service parameter. Leave blank.

env:
  ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
  ALLURE_JOB_RUN_ID: ${{ github.event.inputs.ALLURE_JOB_RUN_ID }}
  ALLURE_ENDPOINT: https://demo.testops.cloud/
  ALLURE_PROJECT_ID: 1
  ALLURE_RESULTS: build/allure-results

jobs:
  all-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up JDK
        uses: actions/setup-java@v3
        with:
          distribution: oracle
          java-version: 17

      - name: Install allurectl
        uses: allure-framework/setup-allurectl@v1

      - name: Build with Gradle
        run: allurectl watch -- ./gradlew clean test

allurectl setup action takes the following parameters:

  • allure-endpoint
    • mandatory parameter, setting the URL address of Allure TestOps instance to upload the test results to.
  • allure-token
    • mandatory parameter used for the authentication of allurectl at Allure TestOps server defined by allure-endpoint parameter
  • allure-project-id
    • mandatory parameter to define the project (numeric ID is used) where launch with test results has to be created
  • allurectl-version
    • optional parameter allowing you to select specific version of allurectl to be used.
    • information on the available releases can be found here.
  • github-token
    • optional parameter to provide alternative GH token

1.4. Run and check the workflow

  1. In GitHub, open the project and go to the workflow run triggered by the latest commit.

  2. If the run is not yet finished, wait until it finishes.

  3. In the run details, click on the job that runs tests.

    Close to the end of its log, there should be a link to the test report at Allure TestOps. Make sure that it is present and works.

    The link to the launch is in the workflow run log.

  4. In the test report at Allure TestOps, open a single test's results.

    At the bottom of the page, there should be a link back to the GitHub workflow run. Make sure that it is present and works.

    The link to the GitHub workflow run is at the bottom of the right pane.

2. Enable triggering GitHub workflows

On the Allure TestOps side, the integration with GitHub needs to be configured on two levels.

First, the administrator specifies the URL of GitHub server.

Then, any project's owner creates an authentication token in GitHub and adds it to Allure TestOps.

2.1. Specify GitHub server in Allure TestOps

  1. Log into Allure TestOps using an administrator account.

  2. Go to Administration → Integrations.

  3. Click + Add integration in the top right corner of the page.

  4. In the dialog that appears, select GitHub.

  5. Fill in the fields:

    • Name — a name to help you recognize the GitHub server, e.g., “GitHub production”.
    • Endpoint — the base URL of GitHub API. For github.com, use “https://api.github.com”. For GitHub Enterprise Server, use “⟨URL⟩/api/v3”, where ⟨URL⟩ is the URL of your GitHub installation.

  6. If you use GitHub Enterprise Server with a self-signed SSL certificate, check the Disable certificate validation checkbox.

  7. Click Add integration.

2.2. Create a token in GitHub

To be able to trigger workflows, Allure TestOps needs to have a personal access token generated by the GitHub server. You can choose the type of token that GitHub will generate: fine-grained or classic (see Managing your personal access tokens in the GitHub documentation).

Instructions for generating a token vary slightly depending on the type.

  1. In GitHub, click on your avatar and go to Settings.

  2. In the menu on the left, click Developer settings.

  3. Go to Personal access tokens → Fine-grained tokens.

  4. Click Generate new token.

  5. Fill in the fields:

    • Token name — a name to help you recognize the token, e.g., “Token for Allure TestOps”.
    • Expiration — specify how long the token should be valid. After this date, the integration will stop working, and you will need to create a new token to continue using the integration.

  6. Under the Repository access section, click Only select repositories. In the dropdown that appears, select the repository containing the workflows that you want to integrate with.

  7. Under the Permissions section, click Repository permissions. In the permission list that appears, find Actions and select “Read and write” next to it.

  8. Click Generate token.

    The page will reload, and the new token will become temporarily visible. Click the Copy icon next to it.

    You will need this token on the next step.

  1. In GitHub, click on your avatar and go to Settings.

  2. In the menu on the left, click Developer settings.

  3. Go to Personal access tokens → Tokens (classic).

  4. Click Generate new token → Generate new token (classic).

  5. Fill in the fields:

    • Note — a name to help you recognize the token, e.g., “Token for Allure TestOps”.
    • Expiration — specify how long the token should be valid. After this date, the integration will stop working, and you will need to create a new token to continue using the integration.

  6. Under the Select scopes section, check “workflow”.

  7. Click Generate token.

    The page will reload, and the new token will become temporarily visible. Click the Copy icon next to it.

    You will need this token on the next step.

2.3. Add the token to an Allure TestOps project

  1. In Allure TestOps, open your project.

  2. Go to Settings → Integrations.

  3. Under the Available integrations, find the GitHub integration and click Add integration next to it.

  4. In the dialog that appears, specify the Token that you got on step 2.2.

    Check that the credentials are correct

    Click Test connection. After a few moments, a message should appear saying “Connection established”.

  5. Click Add integration to close the dialog and save the settings.

3. Parametrize workflows

GitHub uses inputs for passing parameters to workflows' context, i.e. you need to create an input variable which value then will be assigned to an environment variable in the context of your workflow execution and then this value can be used in the tests.

Allure TestOps integrates this feature with its Environment concept, which lets you both set parameters for new jobs, and see parameters set in workflows that were started from CI side.

Despite the tool used to upload (allurectl or a CI plugin) the test results to Allure TestOps, whole pipeline's context (in case of GitHub – the context of a workflow) is transferred to Allure TestOps, and can be used to link environment data to the uploaded test results.

3.1. Declare inputs in the GitHub workflow

To make changes to a workflow, edit its YAML file in the .github/workflows directory of your GitHub repository. You can do it either locally or in the GitHub's web-based text editor.

Add workflow dispatch inputs, their descriptions, and default values into the inputs block. For example:

name: Run tests

on:
  workflow_dispatch:
    inputs:
      ALLURE_JOB_RUN_ID:
        description: ALLURE_JOB_RUN_ID service parameter. Leave blank.
      ALLURE_USERNAME:
        description: ALLURE_USERNAME service parameter. Leave blank.
      TESTS_BROWSER:
        description: Browser to use for tests
        default: chrome
      PRODUCT_VERSION:
        description: Product version
        default: '1.23'
...

and you can use these inputs later in your workflow to pass the information to your tests via environment variables.

        env:
          PRODUCT_VERSION: ${{ github.event.inputs.PRODUCT_VERSION }}
          TESTS_BROWSER: ${{ github.event.inputs.TESTS_BROWSER }}

Among the others environment variables of GitHub workflow these environment variables we just created, will be sent by allurectl to Allure TestOps in scope of a build's context.

3.2. Add global environment variables names

Before using the environment variable data from a GitHub workflow, or generally from any kind of pipeline, you need to create a global environment variable which will store such a data.

  1. Log into Allure TestOps using an administrator account.

  2. Go to Administration → Environment.

  3. For each parameter name that you want to add:

    1. Click + Create.

    2. Enter the name of the new parameter.

    3. Click Submit.

    Environment variable names are added in the Administration settings.

3.3. Map workflow environment variables to global environment variables

After you have an environment variable declared in your workflow (pipeline), and created global environment variable in Allure TestOps, you can start configuring your project environment variables to extract the information you need from workflow's context.

  1. In Allure TestOps, open your project.

  2. Go to Settings → Environment.

  3. For each parameter name that you want to use:

    1. Click + Create if the parameter is not in the list. Otherwise, click the Edit icon next to its name.

    2. In the Mapping key field, specify the name of the environment variable.

    3. In the Environment variable field, select the global parameter name from step 3.2.

    4. Click Submit.

    Mapping environment variables is configured in the project settings.

3.4. Add parameters to the job

  1. In Allure TestOps, open your project.

  2. Go to Jobs.

  3. Find the job which you want to parametrize. Click ⋯ next to the job name, then select Configure.

    The job settings dialog will appear, containing the Parameters section.

  4. For each parameter that you want to add, click Add and fill in the fields:

    • Name — the name of the environment variable (same as the Mapping key from step 3.3).
    • Value — the default value that should be used unless overridden for a specific launch.
    • Environment Variable — the environment variable from step 3.2.

    Multiple environment variables can be specified in the dialog.

  5. Click Submit.

Note that it is recommended to set the same default values both in Allure TestOps and GitHub. This means that a workflow run will get the same environment regardless of what triggered it.

3.5. Executing a workflow with certain environment data

In the previous steps we configured a job on Allure TestOps side, which now has all the parameters we want to pass to our GitHub workflow.

The parameters are listed next to the job name.

Configured job parameters are the ones, you will be able to pass to your pipeline when triggering CI pipelines from Allure TestOps UI.

Parameter rendered as the first one in the list, is the name of an environment variable that we are going to set on CI side, e.g. TEST_BROWSER; the parameter in the parenthesis in the name of a global environment variable which will provide the suggestions for the values, e.g. Browser. And the value (in our example it's "chrome") is the default value we are going to push to a CI system if no alternative value.

To execute tests on certain environment you need to invoke one of the following actions.

Please note, reruns of the tests are always executed on the same environment as it was set during the initial execution.

When you execute any of the said actions, you will see the following form for the creation of a launch.

The “Add” button is at the bottom of the dialog.

You need to click the button saying Add in the Environment section.

Application will add the job parameters to the form and pre-fill these with the job parameters default values.

The parameter values can be entered or selected from lists.

And then you can alter the values you want to pass to the CI pipeline.

If you need to execute your pipeline several times on different sets of the parameters, you need to create a new set of the environment variables and Allure TestOps will request CI to run several pipelines with different sets of parameters.

Examples

This is a very simple workflow, you can use as a start for your GitHub workflow to try the execution of tests and uploading the data to Allure TestOps. It uses pytest as an example, and it contains all the elements required to upload the test results to Allure TestOps and allow the triggering of workflows from Allure TestOps side.

As we mentioned before, there are two ways to add allurectl to the workflow.

  1. using GH action
    • action will download allurectl
    • action will generate all the required parameters based on workflow context or context provided from Allure TestOps
    • action will provide configuration parameters required for the test results upload
  2. using script inside your workflow
    • this option is used in case GH can't be used due to some restrictions
    • script will download allurectl
    • script will generate the required parameters based on workflow context
name: run-and-upload-to-allure-testops

on:
  workflow_dispatch:
    inputs:
      GITHUB_TESTS_ENDPOINT:
        description: "System under test"
        required: true
        default: https://system.under.test
      GITHUB_TESTS_BROWSER:
        description: "Browser to be used in tests"
        required: true
        default: chrome
      ALLURE_JOB_RUN_ID:
        description: "ALLURE_JOB_RUN_ID is service parameter required for triggering workflows from Allure TestOps. Leave blank."
        required: false
      ALLURE_USERNAME:
        description: "ALLURE_USERNAME service parameter. Leave blank"
        required: false


env:
  ALLURE_RESULTS: "allure-results"
  ALLURE_JOB_RUN_ID: ${{ github.event.inputs.ALLURE_JOB_RUN_ID }}

jobs:
  all-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - name: Install dependencies for pytest and allure framework integration
        run: |
          python -m pip install --upgrade pip
          pip install allure-pytest pytest
      - name: install and configure allurectl using GH Action
        uses: allure-framework/setup-allurectl@v1
        with:
          allure-endpoint: https://demo.testops.cloud
          allure-token: ${{ secrets.ALLURE_TOKEN }}
          allure-project-id: 9999
      - name: Run pytest tests
        run: |
          allurectl watch -- pytest ./test --alluredir=${ALLURE_RESULTS} --capture=no || true
        env:
          GITHUB_TESTS_ENDPOINT: ${{ github.event.inputs.GITHUB_TESTS_ENDPOINT }}
          GITHUB_TESTS_BROWSER: ${{ github.event.inputs.GITHUB_TESTS_BROWSER }}
          GITHUB_TESTS_BRANCH: ${{ github.ref_name }}

name: pytest tests with allure framework

on:
  workflow_dispatch:
    inputs:
      GITHUB_TESTS_ENDPOINT:
        description: "System under test"
        required: true
        default: https://system.under.test
      GITHUB_TESTS_BROWSER:
        description: "Browser to be used in tests"
        required: true
        default: chrome
      ALLURE_JOB_RUN_ID:
        description: "Inner parameter for Allure TestOps"
        required: false
      ALLURE_USERNAME:
        description: "ALLURE_USERNAME service parameter. Leave blank"
        required: false

env:
  ALLURE_ENDPOINT: ${{ secrets.ALLURE_ENDPOINT }}
  ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
  ALLURE_PROJECT_ID: ${{ secrets.ALLURE_PROJECT_ID }}
  ALLURE_TESTPLAN_PATH: "./testplan.json"
  ALLURE_RESULTS: "allure-results"
  ALLURE_JOB_RUN_ID: ${{ github.event.inputs.ALLURE_JOB_RUN_ID }}
  GH_BRANCH: ${{ github.ref_name }}
  ALLURE_LAUNCH_TAGS: "github, demo, pytest, $GH_BRANCH"

jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
    - uses: actions/github-script@v4
      id: allure-job-uid
      with:
        result-encoding: string
        script: |
          const result = await github.actions.getWorkflowRun({
            owner: context.repo.owner,
            repo: context.repo.repo,
            run_id: context.runId,
          });
          return `${context.repo.owner}/${context.repo.repo}/actions/workflows/${result.data.workflow_id}`
    - name: Download and make allurectl executable
      run: |
        wget https://github.com/allure-framework/allurectl/releases/download/2.15.1/allurectl_linux_amd64 -O ./allurectl
        chmod +x ./allurectl
    - name: Install dependencies for pytest and allure framework integration
      run: |
        python -m pip install --upgrade pip
        pip install allure-pytest pytest
    - name: Test with pytest and gen allure results
      shell: bash
      working-directory: ${{ github.action_path }}
      env:
        ALLURE_JOB_UID: ${{steps.allure-job-uid.outputs.result}}
        GITHUB_TESTS_ENDPOINT: ${{ github.event.inputs.GITHUB_TESTS_ENDPOINT }}
        GITHUB_TESTS_BROWSER: ${{ github.event.inputs.GITHUB_TESTS_BROWSER }}
      run: |
        ./allurectl watch -- pytest --alluredir=${ALLURE_RESULTS}

Step with id: allure-job-uid is a mandatory one if you are going to trigger the workflows from Allure TestOps.

Troubleshooting

Error 422 when triggering GitHub workflow via Allure TestOps interface

Root cause

Generally, the error 422 is returned by GitHub when you are trying to start workflow using workflow_dispatch, and the parameters defined in the inputs of workflow_dispatch do not match those sent from Allure TestOps.

Expected configuration

As mentioned earlier in the instruction, there are at least two mandatory parameters (with the required attribute set to false) that must be included as inputs in the workflow_dispatch section for Allure TestOps to trigger a workflow:

  • ALLURE_JOB_RUN_ID,
  • ALLURE_USERNAME.

These parameters are set by Allure TestOps automatically without involving any user's actions. Both are necessary for successfully triggering workflows.

If your pipeline (workflow) logic requires additional mandatory data (e.g., browser name, system under test name), these parameters must be configured in Allure TestOps as job configuration parameters.

Example

name: Integration of Allure TestOps with GitHub 

on:
  workflow_dispatch:
    inputs:
      ALLURE_JOB_RUN_ID:
        description: "ALLURE_JOB_RUN_ID service parameter. Leave blank."
        required: false
      ALLURE_USERNAME:
        description: "ALLURE_USERNAME service parameter. Leave blank"
        required: false
      TEST_BROWSER:
        description: "Browser for tests"
        required: true
        default: chrome
      TEST_ENDPOINT:
        description: "System under test"
        required: false
        default: https://staging.system.under.test

For the described configuration of inputs for workflow_dispatch on the Allure TestOps side, you must have the following configuration parameters for the created job:

  1. Branch:

    • Linked to the Branch environment variable (both starting with capital B).
    • This is a special environment variable not explicitly listed in the inputs, but its value is passed to ref parameter in the API call. Therefore, this variable is mandatory (cannot be omitted).

  2. TEST_BROWSER:

    • Passes the browser name to workflow_dispatch.
    • Must be included in the job configuration parameters as it is defined in the workflow_dispatch as required: true.

  3. TEST_ENDPOINT:

    • May be included as an optional job configuration parameter as it is defined in the workflow_dispatch as required: false.
Examples of GitHub behavior when you try executing the configured job
Will work correctly.
All parameters are present in inputs
Will work correctly.
Required inputs (TEST_BROWSER) are present in parameters
Won't work, error 422.
RELEASE_ID is not present in inputs of workflow_dispatch
Won't work, error 422.
Required input TEST_BROWSER is missing

Solution

The only possible solution is to synchronize the configuration parameters of the job created in Allure TestOps with the inputs in workflow_dispatch.

© Qameta Software Inc. All rights reserved.