Command line tool - allurectl

What is allurectl

Allurectl is the command line wrapper of Allure Testops' API allowing you to upload the test results in real time from a build job, and managing entities on Allure Testops side (test cases, launches, projects).

Downloading allurectl

Supported platforms

MacOS on Intel x64 *_darwin_amd64
MacOS on M1 (arm64)*_darwin_arm64
32 bits Linux *_linux_386
64 bits Linux *_linux_amd64
64 bits arm Linux *_linux_arm64
32 bits Windows *_windows_386.exe
64 bits Windows *_windows_amd64.exe
64 bits arm Windows *_windows_arm64.exe

You need to select the binary file suitable for your OS and CPU architecture from the available options.

In example below we download linux application for x64 CPU:

wget https://github.com/allure-framework/allurectl/releases/latest/download/allurectl_linux_amd64 -O ./allurectl
chmod +x ./allurectl

allurectl working modes

There are two modes:

Non-CI mode
CI mode

Non-CI mode

After allurectl started, it checks whether CI specific variables are defined and if they are absent, the data upload is considered as local upload (from PC). In this mode you cannot use/manipulate parameters related to job runs.

Passing parameters to allurectl

There are 2 ways how you can send the parameters to allurectl:

Using command line parameters.
- This option is good when you need one-time upload for some of your tests.
Using environment variables.
- This option is better when you upload your data on regular basis, it is allowing you to store and reuse important parameters, so you don't need to use them in the command line every time.

Upload using command line parameters

allurectl upload --endpoint https://allure.company.com \
    --token 55555555-5555-5555-5555-555555555555 \
    --project-id 100 \
    --launch-name "Local PC manual launch 2200-12-31" \
    path/to/allure-results

Upload using environment variables

# Define environment variables
export ALLURE_ENDPOINT=https://allure.company.com
export ALLURE_TOKEN=55555555-5555-5555-5555-555555555555
export ALLURE_PROJECT_ID=100
export ALLURE_LAUNCH_NAME="Local PC manual launch 2200-12-31"
export ALLURE_LAUNCH_TAGS="release, critical"
# Run upload process somewhere
allurectl upload path/to/allure-results

What will happen

Now, what will happen if you used one of the options above:

A new launch with the name Local PC manual launch 2200-12-31 will be created on Allure Testops side
A new session for test results upload will be created
Test results from path/to/allure-results folder will be sent to Allure Testops' launch Local PC manual launch 2200-12-31 in scope of the created session.
Session will be closed.
Launch will remain open until manually closed or closed base on the automatic closure rule defined for the project 100.

CI mode

Each CI system has its own set of environment variables identifying it. If allurectl detects such variables, then the launch is considered as a launch from an CI system.

For a CI upload the following additional CI specific information will be included into upload and available in Allure Testops related to the communications between allurectl and Allure Testops.

allurectl also uses some internal variables to send the information to a proper launch on Allure Testops side.

Allure Testops API token

Before you will be able to use allurectl, you need to generate Allure Testops secret token in your profile, so allurectl will be able to be authenticated on Allure Testops side.

The process of getting of the API token is described in Allure Testops documentation.

allurectl watch workflow

Generally, allurectl watch does the same things which allurectl upload does with one important difference - watch command allows sending the test result in real time, i.e. you don't need to wait till all the tests are completed and that will decrease the workload and hasten the test results processing on Allure Testops side.

allurectl watch is a wrapper for your test execution, i.e. you need to provide the command which you are using to execute your tests to allurectl watch.

The usage

As usual you need to provide parameters needed for the connection to Allure Testops server (in the example below these are the environment variables, and we strongly recommend you using the environment variables to pass these parameters to allurectl), then you show allurectl where you're expecting the test results to appear and then provide the command to start tests execution.

export ALLURE_ENDPOINT=https://allure.company.com
export ALLURE_TOKEN=55555555-5555-5555-5555-555555555555
export ALLURE_PROJECT_ID=100
export ALLURE_LAUNCH_NAME="Hello, world" # you can use here the environment variables of your job/pipeline
export ALLURE_LAUNCH_TAGS="release, critical"

allurectl watch --results path/to/allure-results -- ./gradlew clean test

alternatively, you can provide all the start parameters in the environment variables, so the watch will look prettier and concise:

export ALLURE_ENDPOINT=https://allure.company.com
export ALLURE_TOKEN=542dcd56-b0e2-4fdd-8ecf-bacf0f33d505
export ALLURE_PROJECT_ID=12
export ALLURE_LAUNCH_NAME="Hello, world"
export ALLURE_RESULTS=path/to/allure-results
export ALLURE_LAUNCH_TAGS="release, critical"

allurectl watch -- ./gradlew clean test

allurectl upload workflow

The applicability

Use allurectl with the upload command after all your tests run. We do not recommend using allurectl upload as background process.

We recommend the usage of allurectl watch to send the data from CI. Use upload only in case watch is not acceptable for you.

The settings

The following environment variables could be set for easier usage of allurectl, this is an alternative to setting same parameters in the command line. Using the variables makes the execution of allurectl more clear and leaves less chance for the human error.

Env variable	Comment
ALLURE_ENDPOINT	URL of Allure Testops server
ALLURE_PROJECT_ID	the ID of a project in Allure Testops, this is the 1st column of Allure Testops main screen
ALLURE_TOKEN	the user's personal access token generated in the user's profile in section API Tokens!
ALLURE_LAUNCH_NAME	Name of the launch which will be displayed in Allure Testops UI
ALLURE_LAUNCH_TAGS	Comma separated list of tags which will be displayed in Allure Testops UI

Please refer to your CI settings details to set up allurectl environments variables.

Tests rerun and selective run with allurectl

The most important thing with test rerun and selective run is the test plan. In this particular case when we're talking about the selective run and rerun (rerun is actually is a special case for selective run) the test plan is a file (specifically, it's testplan.json) with the list of test cases, that your test framework needs to run.

Now, let's discuss how this integration works.

Selective tests run integration

On Allure Testops side we create a list of test cases we need to (re)run on CI side. Each test case is identified by AllureID and a selector. Selector is the way test framework identifies certain test, generally it's test framework dependent.
The list of test cases is then saved to CI into testplan.json file.
On CI side we create an environment variable ALLURE_TESTPLAN_PATH and set the value to the path to testplan.json file
When Allure Framework adaptor finds ALLURE_TESTPLAN_PATH is available, then it tries to read testplan.json.
If the file testplan.json is successfully read, then the integration instructs the test framework to run only the tests specified in the testplan.json.

If CI starts all tests from your project, then either something is not configured on your side or your test framework has no integration with Allure Testops. If you are facing this, please file a support (not a bug) request to our technical support.

testplan.json structure

The testplan.json file looks like follows.

Knowing this structure, you can create testplan.json on your local PC, initialize the environment variable ALLURE_TESTPLAN_PATH with path to testplan.json and run your tests locally without any additional filters in the same session, if only tests from testplan.json will run, then you have the working integration for selective run, otherwise you need to configure the integration or develop it for your test framework.


{
  "version":"1.0",
  "tests": [
    {
      "id": "10",
      "selector": "io.qameta.allure.examples.junit4.AllureSimpleTest.allureSimpleTest"
    },
    {
      "id": "11",
      "selector": "io.qameta.allure.examples.junit4.AllureParameterizedTest.allureParameterizedTest[First Name]"
    }
  ]
}

where

id is the test ID (Allure ID) from Allure Testops
selector is the alternative ID which is equal to test's full path by default.
- we're planning to extend the information provided in the selector

Saving the test plan to CI, run tests and upload the tests results

In all CIs we have the same sequence:

Allure Testops starts a Job and adds ALLURE_JOB_RUN_ID =
to the job.
- if allurectl detects ALLURE_JOB_RUN_ID then it ignores all other variables an session is created in the Job run set by Allure Testops.
Allure Testops provides the environment variable ALLURE_TESTPLAN_PATH: with path to testplan.json to the job run.
allurectl creates testplan.json and fills it with the information about the list of the tests.
allurectl executes your tests and makes the index list of the test results files and send the tests results files to Allure Testops using watch command
- allurectl watch -- ./gradlew clean test

All work with ALLURE_TESTPLAN_PATH and testplan.json is fully managed by allurectl. No user involvement and settings are required as of the allurectl release 2.10.0

Running nested jobs

It happens you need to run several jobs on a CI and send all the test results from several jobs into one aggregated launch on Allure Testops side.

ALLURE_JOB_RUN_ID environment variable to the rescue!

If allurectl detects that ALLURE_JOB_RUN_ID is available, then it does not create new entities but uses the one from ALLURE_JOB_RUN_ID. Same way works the execution of tests from Allure Testops side.

The workflow

Create aggregating job on CI side
Create new job run in the aggregating job and store the content of created ALLURE_JOB_RUN_ID
Create nested/child job, pass the environment variable ALLURE_JOB_RUN_ID and its content to the nested/child job.
- all the test results from the jobs with the same ALLURE_JOB_RUN_ID will be transferred to the same Launch in Allure Testops.
allurectl stops the job run (no additional command is needed).

allurectl cookbook

Passing the parameters to allurectl

Environment variables

The following environment variables need to be set for easier usage of allurectl

Env variable	Comment
ALLURE_ENDPOINT	URL of Allure Testops server
ALLURE_PROJECT_ID	the ID of a project in Allure Testops, this is the 1st column of Allure Testops main screen
ALLURE_TOKEN	the user's personal access token generated in the user's profile in section API Tokens!
ALLURE_LAUNCH_NAME	Name of the launch that will be displayed in Allure Testops UI

Test connection

This needs to be used for connection check purposes only. Do not use this command in your pipelines to authenticate the test results upload process. The authentication for test results upload routines is done otherwise (see below).

This could be done from your local computer or in a CI pipeline (see remark above).

To check the connection to Allure Testops instance you can use the following command:

export ALLURE_TOKEN=<API-TOKEN>
export ALLURE_ENDPOINT=https://demo.testops.cloud
allurectl auth login

Alternatively, you can use CL switches, but we recommend using the environment variables.

For more information use the allurectl auth login --help command.

Upload the test results to Allure Testops

There are 2 options.

usage of watch routine
usage of upload routine

watch

watch is main recommended workflow for test results upload.

allurectl watch is a wrapper for your test execution, i.e. you need to provide the command which you are using to execute your tests to allurectl watch.

The usage

export ALLURE_ENDPOINT=https://allure.company.com
export ALLURE_TOKEN=55555555-5555-5555-5555-555555555555
export ALLURE_PROJECT_ID=100
export ALLURE_LAUNCH_NAME="Hello, world" # you can use here the environment variables of your job/pipeline
...
allurectl watch --results path/to/allure-results -- ./gradlew clean test

alternatively, you can provide all the start parameters in the environment variables, so the watch will look prettier and concise:

export ALLURE_ENDPOINT=https://allure.company.com
export ALLURE_TOKEN=542dcd56-b0e2-4fdd-8ecf-bacf0f33d505
export ALLURE_PROJECT_ID=12
export ALLURE_LAUNCH_NAME="Hello, world"
export ALLURE_RESULTS=path/to/allure-results
...
allurectl watch -- ./gradlew clean test

upload

We recommend using allurectl watch to send the data from CI. Use upload only in case watch is not acceptable for you.

upload workflow is only applicable when the test results are available after all tests execution (there are such test frameworks, not many but they exist) or you need to upload a directory with the test results you have.

Use allurectl with the upload command after all your tests run. We do not recommend using allurectl upload as background process.

Upload using command line parameters

allurectl upload --endpoint https://allure.company.com \
    --token 55555555-5555-5555-5555-555555555555 \
    --project-id 100 \
    --launch-name "Local PC manual launch 2200-12-31" \
    path/to/allure-results

Upload using environment variables

# Define environment variables
export ALLURE_ENDPOINT=https://demo.testops.cloud
export ALLURE_TOKEN=55555555-5555-5555-5555-555555555555
export ALLURE_PROJECT_ID=100

# Run upload process somewhere
allurectl upload --launch-name "Local PC manual launch 2200-12-31" path/to/allure-results

Getting Allure Testops launch information

The task

We want to get the information on a launch created after the execution of watch or upload workflows to pass the information to chat, email message etc.

How to

This works in CI mode only

The information on the entities created on Allure Testops side can be placed to the environment variables and then used by invoking of the following sequence of the commands:

#define env vars
export ALLURE_ENDPOINT=https://demo.testops.cloud
export ALLURE_TOKEN=<toke>
export ALLURE_PROJECT_ID=<ID>
export ALLURE_LAUNCH_NAME="Hello, world"
export ALLURE_RESULTS=path/to/allure-results
# exec the tests
allurectl watch -- ./gradlew clean test

export $(allurectl job-run env)
# this will just show the list of all ENV vars related to allurectl execution
printenv | grep ALLURE_

printenv | grep ALLURE_ will result in the following output

ALLURE_ENDPOINT=https://demo.testops.cloud
ALLURE_LAUNCH_ID=11111
ALLURE_RESULTS=./allure-results
ALLURE_JOB_RUN_ID=12345
ALLURE_LAUNCH_TAGS=master, gitlab, demo, pytest, skip-live-doc, ignore
ALLURE_TOKEN=[MASKED]
ALLURE_LAUNCH_NAME=allure-pytest - 1ea04f48
ALLURE_JOB_RUN_URL=https://demo.testops.cloud/jobrun/14433
ALLURE_LAUNCH_URL=https://demo.testops.cloud/launch/31897
ALLURE_PROJECT_ID=433

To provide the link to the created launch you can use either ALLURE_JOB_RUN_URL or ALLURE_LAUNCH_URL.

ALLURE_JOB_RUN_URL is an entity (there could be N job-runs) inside a launch, so if you merge two or more launches in one, then ALLURE_JOB_RUN_URL will always point to a correct launch.

Creating a launch based on test plan created in Allure Testops project

The task

End user wants to create a test plan on Allure Testops side, and then execute only the tests cases by referencing the test plan in the pipeline.

How to

The task can be done by creation of testplan.json file before executing of allurectl watch

export ALLURE_TOKEN=<token>
export ALLURE_ENDPOINT=https://demo.testops.cloud
export ALLURE_PROJECT_ID=111
export ALLURE_LAUNCH_NAME="$(date "+%Y-%m-%d %H%M%S") executing test plan"
export ALLURE_LAUNCH_TAGS="watch, testplan"
export ALLURE_RESULTS="allure-results"
export ALLURE_TESTPLAN_PATH="testplan.json"
# create tetsplan.json based on AQL for test cases "testPlan=222" where 222 is the ID of a test plan
./allurectl test-case plan -q "testPlan=222" --output-file ${ALLURE_TESTPLAN_PATH}
# execute test cases based on testplan.json
./allurectl watch -- [tests_execution_command]

Test result upload debugging

The task

I want to see the debug information on my upload process to troubleshoot the problems with the test results upload.

How to

Detailed debug is available starting from allurectl release 2.11.0

allurectl has special command line switch --log which allows collecting logs of the test results upload.

./allurectl --log debug upload ${ALLURE_RESULTS}
./allurectl --log debug watch

the --log switch should be added before watch/upload command, if added after then switch will be ignored.

For each uploaded batch of files allurectl will generate line with the details on the upload process.

Logging levels

There are following logging levels from less informative to overwhelmingly informative.

fatal
error
warn
info
debug
trace

How to automatically add an issue link to a launch

The task

I want to automatically add issue links to launches every time a launch is created from CI job.

How to

Adding issue links is available starting from allurectl release 2.2.0

Non CI mode:

For non-CI mode, you need to know the exact ID of a launch, to attach issue information to.

allurectl launch add-issue --launch-id launch_id --integration-id integration_id --issue-keys issue_key_1,issue_key_2,...,issue_key_N

For CI mode(in pipeline after using watch command):

allurectl launch add-issue --current-launch --integration-id integration_id --issue-keys issue_key_1,issue_key_2,...,issue_key_N

where:

launch_id - must be your launch id
integration_id - must be your integration id (you can find this ID in Allure Testops UI)
issue_key - must be your issue key (for example AE-1,AE-2)

Executing tests in a docker container or on a remote VM/bare metal server

Task

My tests need to be executed outside the pipeline on a remote VM or in a docker container. I need the test results to be uploaded to Allure Testops in a launch linked to the pipeline which initiated the tests execution.

How to

Feature is available starting from allurectl release 2.10.0

allurectl allows to pack the pipeline's context and pass this information to other system, and then unpack the context in the system where your test are supposed to be executed, so allurectl in VM or in a docker container will think it's running inside a pipeline.

Pack the context

Action to be performed in the pipeline which is responsible for the triggering of the tests execution.

You need to execute allurectl with parameters ci env --encode in the scope of the parent pipeline and save the results of the command execution as ALLURE_CI_ENV environment variable.

ALLURE_CI_ENV=$(allurectl ci env --encode)

Pass the context

You need to pass the content of ALLURE_CI_ENV to the place where your tests are to be executed. This variable needs to be created and the value set before allurectl is executed.

After the variable is created and its value set, you can execute your tests by allurectl as usual.

allurectl watch -- exec-your-tests-here

The upload of test results from VM/container will be linked to the pipeline which triggered the tests executions.

Executing tests even if Allure Testops server is not reachable

Normally, if allurectl cannot connect to Allure Testops server, it does not start the execution of the tests and fails the pipeline.

The task

I want my tests to be executed even if Allure Testops server is not reachable du to any reason (server is in the maintenance, server is down, no route to the host, network issues, etc.).

How to

Feature is available starting from the allurectl release 2.15.0

To prevent allurectl from failing the pipeline, we introduced the CL switch --silent. When using this switch, allurectl will check the server's availability, if it is not reachable, then allurectl will raise a warning in the console log, and start executing the tests as per provided CLI command without attempting to upload the test results.

Example

allurectl watch --silent -- [tests_execution_command]

Avoiding the upload of big *-result.json files which won't be processed by Allure Testops

Feature is available starting from the allurectl release 2.15.0

Allure Testops has limitation on the size of the *-result.json files. The size limit is set to 2,000,000 bytes (two million) bytes. If the result file exceeds the limit, then it considered as an attachment and is not processed by the parser, hence the result won't be saved to the database and won't appear in the UI. If no link to his file has been found in other *-result.json files, then it will be deleted as soon as a launch is closed.

The task

I want to avoid uploading files, which won't be processed to save time and resources for the upload and processing.

How to

We introduced CL switch --skip-too-big which will prevent uploading of the files exceeding the size of 2,000,000 bytes. The tool won't upload such files and will raise the warning in the stdout containing the information that certain files are too big to be processed by the server.

Example

allurectl watch --skip-too-big  -- [tests_execution_command]

Unload the storage by skipping attachments for successfully passed tests

The task

I want to unload the storage and hasten the results processing by avoiding the upload of attachments for successfully passed tests as these are no interest for us, we just need failed or broken tests to have the attachments.

How to #1

We introduced CL switch --ignore-passed-test-attachments in allurectl 2.9.0 which allows doing it.

This will work for the attachments in test's body. The attachments in fixtures (setup/tear-down blocks) will be uploaded even if --ignore-passed-test-attachments is specified.

Example #1

./allurectl watch --ignore-passed-test-attachments

or for the upload routine

./allurectl upload --ignore-passed-test-attachments ${ALLURE_RESULTS}

How to #2

There is ac CLI switch --exclude-files that allows you ignoring files by supplying a regular expression for files names.

The regexp to be used, has Golang flavour, so you're better off checking with something like regex101.com before applying the ignore file filers.

Example #2

This will ignore all -container.json (setup/tear-down blocks of your tests) files in allure-results directory.

./allurectl watch --exclude-files /*-container.json

or for the upload routine

./allurectl upload --exclude-files /*-container.json ${ALLURE_RESULTS}

Releases

Information about allurectl releases is available on GitHub