Command line tool - allurectl

What is allurectl

Allurectl is the command line wrapper of Allure TestOps' API allowing you to upload the test results in real time from a build job, and managing entities on Allure TestOps side (test cases, launches, projects).

Downloading allurectl

Supported platforms

MacOS on Intel x64 *_darwin_amd64
MacOS on M1 (arm64)*_darwin_arm64
32 bits Linux *_linux_386
64 bits Linux *_linux_amd64
64 bits arm Linux *_linux_arm64
32 bits Windows *_windows_386.exe
64 bits Windows *_windows_amd64.exe
64 bits arm Windows *_windows_arm64.exe

You need to select the binary file suitable for your OS and CPU architecture from the available options.

In example below we download linux application for x64 CPU:

wget https://github.com/allure-framework/allurectl/releases/latest/download/allurectl_linux_amd64 -O ./allurectl
chmod +x ./allurectl

allurectl working modes

There are two modes:

Non-CI mode
CI mode

Non-CI mode

After allurectl started, it checks whether CI specific variables are defined and if they are absent, the data upload is considered as local upload (from PC). In this mode you cannot use/manipulate parameters related to job runs.

Passing parameters to allurectl

There are 2 ways how you can send the parameters to allurectl:

Using command line parameters.
- This option is good when you need one-time upload for some of your tests.
Using environment variables.
- This option is better when you upload your data on regular basis, it is allowing you to store and reuse important parameters, so you don't need to use them in the command line every time.

Upload using command line parameters

allurectl upload --endpoint https://allure.company.com \
    --token 55555555-5555-5555-5555-555555555555 \
    --project-id 100 \
    --launch-name "Local PC manual launch 2200-12-31" \
    path/to/allure-results

Upload using environment variables

# Define environment variables
export ALLURE_ENDPOINT=https://allure.company.com
export ALLURE_TOKEN=55555555-5555-5555-5555-555555555555
export ALLURE_PROJECT_ID=100
export ALLURE_LAUNCH_NAME="Local PC manual launch 2200-12-31"
export ALLURE_LAUNCH_TAGS="release, critical"
# Run upload process somewhere
allurectl upload path/to/allure-results

What will happen

Now, what will happen if you used one of the options above:

A new launch with the name Local PC manual launch 2200-12-31 will be created on Allure TestOps side
A new session for test results upload will be created
Test results from path/to/allure-results folder will be sent to Allure TestOps' launch Local PC manual launch 2200-12-31 in scope of the created session.
Session will be closed.
Launch will remain open until manually closed or closed base on the automatic closure rule defined for the project 100.

CI mode

Each CI system has its own set of environment variables identifying it. If allurectl detects such variables, then the launch is considered as a launch from an CI system.

For a CI upload the following additional CI specific information will be included into upload and available in Allure TestOps related to the communications between allurectl and Allure TestOps.

allurectl also uses some internal variables to send the information to a proper launch on Allure TestOps side.

Allure TestOps API token

Before you will be able to use allurectl, you need to generate Allure TestOps secret token in your profile, so allurectl will be able to be authenticated on Allure TestOps side.

The process of getting of the API token is described in Allure TestOps documentation.

allurectl watch workflow

Generally, allurectl watch does the same things which allurectl upload does with one important difference - watch command allows sending the test result in real time, i.e. you don't need to wait till all the tests are completed and that will decrease the workload and hasten the test results processing on Allure TestOps side.

allurectl watch is a wrapper for your test execution, i.e. you need to provide the command which you are using to execute your tests to allurectl watch.

The usage

As usual you need to provide parameters needed for the connection to Allure TestOps server (in the example below these are the environment variables, and we strongly recommend you using the environment variables to pass these parameters to allurectl), then you show allurectl where you're expecting the test results to appear and then provide the command to start tests execution.

export ALLURE_ENDPOINT=https://allure.company.com
export ALLURE_TOKEN=55555555-5555-5555-5555-555555555555
export ALLURE_PROJECT_ID=100
export ALLURE_LAUNCH_NAME="Hello, world" # you can use here the environment variables of your job/pipeline
export ALLURE_LAUNCH_TAGS="release, critical"

allurectl watch --results path/to/allure-results -- ./gradlew clean test

alternatively, you can provide all the start parameters in the environment variables, so the watch will look prettier and concise:

export ALLURE_ENDPOINT=https://allure.company.com
export ALLURE_TOKEN=542dcd56-b0e2-4fdd-8ecf-bacf0f33d505
export ALLURE_PROJECT_ID=12
export ALLURE_LAUNCH_NAME="Hello, world"
export ALLURE_RESULTS=path/to/allure-results
export ALLURE_LAUNCH_TAGS="release, critical"

allurectl watch -- ./gradlew clean test

allurectl upload workflow

The applicability

Use allurectl with the upload command after all your tests run. We do not recommend using allurectl upload as background process.

We recommend the usage of allurectl watch to send the data from CI. Use upload only in case watch is not acceptable for you.

The settings

The following environment variables could be set for easier usage of allurectl, this is an alternative to setting same parameters in the command line. Using the variables makes the execution of allurectl more clear and leaves less chance for the human error.

Env variable	Comment
ALLURE_ENDPOINT	URL of Allure TestOps server
ALLURE_PROJECT_ID	the ID of a project in Allure TestOps, this is the 1st column of Allure TestOps main screen
ALLURE_TOKEN	the user's personal access token generated in the user's profile in section API Tokens!
ALLURE_LAUNCH_NAME	Name of the launch which will be displayed in Allure TestOps UI
ALLURE_LAUNCH_TAGS	Comma separated list of tags which will be displayed in Allure TestOps UI

Please refer to your CI settings details to set up allurectl environments variables.

Tests rerun and selective run with allurectl

The most important thing with test rerun and selective run is the test plan. In this particular case when we're talking about the selective run and rerun (rerun is actually is a special case for selective run) the test plan is a file (specifically, it's testplan.json) with the list of test cases, that your test framework needs to run.

Now, let's discuss how this integration works.

Selective tests run integration

Starting from allurectl release 2.10.x, the tool and Allure TestOps fully control the process of selective test run (please mind the Allure Framework part as well), and it is not required to define ALLURE_TESTPLAN_PATH and executing job-run plan command in your pipeline. You can still use the instructions described below. However, we recommend leaving this whole process to Allure TestOps and allurectl.

On Allure TestOps side we create a list of test cases we need to (re)run on CI side. Each test case is identified by AllureID and a selector. Selector is the way test framework identifies certain test, generally it's test framework dependent.
The list of test cases is then saved to CI into testplan.json file.
On CI side we create an environment variable ALLURE_TESTPLAN_PATH and set the value to the path to testplan.json file
When Allure Framework adaptor finds ALLURE_TESTPLAN_PATH is available, then it tries to read testplan.json.
If the file testplan.json is successfully read, then the integration instructs the test framework to run only the tests specified in the testplan.json.

If CI starts all tests from your project, then either something is not configured on your side or your test framework has no integration with Allure TestOps. If you are facing this, please file a support (not a bug) request to our technical support.

testplan.json structure

The testplan.json file looks like follows.

Knowing this structure, you can create testplan.json on your local PC, initialize the environment variable ALLURE_TESTPLAN_PATH with path to testplan.json and run your tests locally without any additional filters in the same session, if only tests from testplan.json will run, then you have the working integration for selective run, otherwise you need to configure the integration or develop it for your test framework.


{
  "version":"1.0",
  "tests": [
    {
      "id": "10",
      "selector": "io.qameta.allure.examples.junit4.AllureSimpleTest.allureSimpleTest"
    },
    {
      "id": "11",
      "selector": "io.qameta.allure.examples.junit4.AllureParameterizedTest.allureParameterizedTest[First Name]"
    }
  ]
}

where

id is the test ID (Allure ID) from Allure TestOps
selector is the alternative ID which is equal to test's full path by default.

Saving the test plan to CI, run tests and upload the tests results

In all CIs we have the same sequence:

Allure TestOps starts a Job and adds ALLURE_JOB_RUN_ID =
to the job.
- if allurectl detects ALLURE_JOB_RUN_ID then it ignores all other variables an session is created in the Job run set by Allure TestOps.
Allure TestOps provides the environment variable ALLURE_TESTPLAN_PATH: with path to testplan.json to the job run.
allurectl creates testplan.json and fills it with the information about the list of the tests.
allurectl executes your tests and makes the index list of the test results files and send the tests results files to Allure TestOps using watch command
- allurectl watch -- ./gradlew clean test

All work with ALLURE_TESTPLAN_PATH and testplan.json is fully managed by allurectl. No user involvement and settings are required as of the allurectl release 2.10.0

Running nested jobs

It happens you need to run several jobs on a CI and send all the test results from several jobs into one aggregated launch on Allure TestOps side.

ALLURE_JOB_RUN_ID environment variable to the rescue!

If allurectl detects that ALLURE_JOB_RUN_ID is available, then it does not create new entities but uses the one from ALLURE_JOB_RUN_ID. Same way works the execution of tests from Allure TestOps side.

The workflow

Create aggregating job on CI side
Create new job run in the aggregating job and store the content of created ALLURE_JOB_RUN_ID
Create nested/child job, pass the environment variable ALLURE_JOB_RUN_ID and its content to the nested/child job.
- all the test results from the jobs with the same ALLURE_JOB_RUN_ID will be transferred to the same Launch in Allure TestOps.
allurectl stops the job run (no additional command is needed).

allurectl cookbook

For each command, the full list of supported options is available with the --help command line switch. For example: allurectl watch --help.

If you have any doubts, please contact support.

Passing the parameters to allurectl

Environment variables

The following environment variables need to be set for easier usage of allurectl

Env variable	Comment
ALLURE_ENDPOINT	URL of Allure TestOps server
ALLURE_PROJECT_ID	the ID of a project in Allure TestOps, this is the 1st column of Allure TestOps main screen
ALLURE_TOKEN	the user's personal access token generated in the user's profile in section API Tokens!
ALLURE_LAUNCH_NAME	Name of the launch that will be displayed in Allure TestOps UI

Test connection

This needs to be used for connection check purposes only. Do not use this command in your pipelines to authenticate the test results upload process. The authentication for test results upload routines is done otherwise (see below).

This could be done from your local computer or in a CI pipeline (see remark above).

To check the connection to Allure TestOps instance you can use the following command:

export ALLURE_TOKEN=<API-TOKEN>
export ALLURE_ENDPOINT=https://demo.testops.cloud
allurectl auth login

Alternatively, you can use CL switches, but we recommend using the environment variables.

For more information use the allurectl auth login --help command.

Upload the test results to Allure TestOps

There are 2 options.

usage of watch routine
usage of upload routine

watch

watch is main recommended workflow for test results upload.

allurectl watch is a wrapper for your test execution, i.e. you need to provide the command which you are using to execute your tests to allurectl watch.

The usage

export ALLURE_ENDPOINT=https://allure.company.com
export ALLURE_TOKEN=55555555-5555-5555-5555-555555555555
export ALLURE_PROJECT_ID=100
export ALLURE_LAUNCH_NAME="Hello, world" # you can use here the environment variables of your job/pipeline
...
allurectl watch --results path/to/allure-results -- ./gradlew clean test

alternatively, you can provide all the start parameters in the environment variables, so the watch will look prettier and concise:

export ALLURE_ENDPOINT=https://allure.company.com
export ALLURE_TOKEN=542dcd56-b0e2-4fdd-8ecf-bacf0f33d505
export ALLURE_PROJECT_ID=12
export ALLURE_LAUNCH_NAME="Hello, world"
export ALLURE_RESULTS=path/to/allure-results
...
allurectl watch -- ./gradlew clean test

upload

We recommend using allurectl watch to send the data from CI. Use upload only in case watch is not acceptable for you.

upload workflow is only applicable when the test results are available after all tests execution (there are such test frameworks, not many but they exist) or you need to upload a directory with the test results you have.

Use allurectl with the upload command after all your tests run. We do not recommend using allurectl upload as background process.

Upload using command line parameters

allurectl upload --endpoint https://allure.company.com \
    --token 55555555-5555-5555-5555-555555555555 \
    --project-id 100 \
    --launch-name "Local PC manual launch 2200-12-31" \
    path/to/allure-results

Upload using environment variables

# Define environment variables
export ALLURE_ENDPOINT=https://demo.testops.cloud
export ALLURE_TOKEN=55555555-5555-5555-5555-555555555555
export ALLURE_PROJECT_ID=100

# Run upload process somewhere
allurectl upload --launch-name "Local PC manual launch 2200-12-31" path/to/allure-results

Getting Allure TestOps launch information

The task

We want to get the information on a launch created after the execution of watch or upload workflows to pass the information to chat, email message etc.

How to

This works in CI mode only

The information on the entities created on Allure TestOps side can be placed to the environment variables and then used by invoking of the following sequence of the commands:

#define env vars
export ALLURE_ENDPOINT=https://demo.testops.cloud
export ALLURE_TOKEN=<toke>
export ALLURE_PROJECT_ID=<ID>
export ALLURE_LAUNCH_NAME="Hello, world"
export ALLURE_RESULTS=path/to/allure-results
# exec the tests
allurectl watch -- ./gradlew clean test

export $(allurectl job-run env)
# this will just show the list of all ENV vars related to allurectl execution
printenv | grep ALLURE_

printenv | grep ALLURE_ will result in the following output

ALLURE_ENDPOINT=https://demo.testops.cloud
ALLURE_LAUNCH_ID=11111
ALLURE_RESULTS=./allure-results
ALLURE_JOB_RUN_ID=12345
ALLURE_LAUNCH_TAGS=master, gitlab, demo, pytest, skip-live-doc, ignore
ALLURE_TOKEN=[MASKED]
ALLURE_LAUNCH_NAME=allure-pytest - 1ea04f48
ALLURE_JOB_RUN_URL=https://demo.testops.cloud/jobrun/14433
ALLURE_LAUNCH_URL=https://demo.testops.cloud/launch/31897
ALLURE_PROJECT_ID=433

To provide the link to the created launch you can use either ALLURE_JOB_RUN_URL or ALLURE_LAUNCH_URL.

ALLURE_JOB_RUN_URL is an entity (there could be N job-runs) inside a launch, so if you merge two or more launches in one, then ALLURE_JOB_RUN_URL will always point to a correct launch.

Creating a launch based on test plan created in Allure TestOps project

The task

End user wants to create a test plan on Allure TestOps side, and then execute only the tests cases by referencing the test plan in the pipeline.

How to

The task can be done by creation of testplan.json file before executing of allurectl watch

export ALLURE_TOKEN=<token>
export ALLURE_ENDPOINT=https://demo.testops.cloud
export ALLURE_PROJECT_ID=111
export ALLURE_LAUNCH_NAME="$(date "+%Y-%m-%d %H%M%S") executing test plan"
export ALLURE_LAUNCH_TAGS="watch, testplan"
export ALLURE_RESULTS="allure-results"
export ALLURE_TESTPLAN_PATH="testplan.json"
# create tetsplan.json based on AQL for test cases "testPlan=222" where 222 is the ID of a test plan
./allurectl test-case plan -q "testPlan=222" --output-file ${ALLURE_TESTPLAN_PATH}
# execute test cases based on testplan.json
./allurectl watch -- [tests_execution_command]

Test result upload debugging

The task

I want to see the debug information on my upload process to troubleshoot the problems with the test results upload.

How to

Detailed debug is available starting from allurectl release 2.11.0

allurectl has special command line switch --log which allows collecting logs of the test results upload.

./allurectl --log debug upload ${ALLURE_RESULTS}
./allurectl --log debug watch

the --log switch should be added before watch/upload command, if added after then switch will be ignored.

For each uploaded batch of files allurectl will generate line with the details on the upload process.

Logging levels

There are following logging levels from less informative to overwhelmingly informative.

fatal
error
warn
info
debug
trace

How to automatically add an issue link to a launch

The task

I want to automatically add issue links to launches every time a launch is created from CI job.

How to

Adding issue links is available starting from allurectl release 2.2.0

Non CI mode:

For non-CI mode, you need to know the exact ID of a launch, to attach issue information to.

allurectl launch add-issue --launch-id launch_id --integration-id integration_id --issue-keys issue_key_1,issue_key_2,...,issue_key_N

For CI mode(in pipeline after using watch command):

allurectl launch add-issue --current-launch --integration-id integration_id --issue-keys issue_key_1,issue_key_2,...,issue_key_N

where:

launch_id - must be your launch id
integration_id - must be your integration id (you can find this ID in Allure TestOps UI)
issue_key - must be your issue key (for example AE-1,AE-2)

Executing tests in a docker container or on a remote VM/bare metal server

Task

My tests need to be executed outside the pipeline on a remote VM or in a docker container. I need the test results to be uploaded to Allure TestOps in a launch linked to the pipeline which initiated the tests execution.

How to

Feature is available starting from allurectl release 2.10.0

allurectl allows to pack the pipeline's context and pass this information to other system, and then unpack the context in the system where your test are supposed to be executed, so allurectl in VM or in a docker container will think it's running inside a pipeline.

Pack the context

Action to be performed in the pipeline which is responsible for the triggering of the tests execution.

You need to execute allurectl with parameters ci env --encode in the scope of the parent pipeline and save the results of the command execution as ALLURE_CI_ENV environment variable.

ALLURE_CI_ENV=$(allurectl ci env --encode)

Pass the context

You need to pass the content of ALLURE_CI_ENV to the place where your tests are to be executed. This variable needs to be created and the value set before allurectl is executed.

After the variable is created and its value set, you can execute your tests by allurectl as usual.

allurectl watch -- exec-your-tests-here

The upload of test results from VM/container will be linked to the pipeline which triggered the tests executions.

Executing tests even if Allure TestOps server is not reachable

Normally, if allurectl cannot connect to Allure TestOps server, it does not start the execution of the tests and fails the pipeline.

The task

I want my tests to be executed even if Allure TestOps server is not reachable du to any reason (server is in the maintenance, server is down, no route to the host, network issues, etc.).

How to

Feature is available starting from the allurectl release 2.15.0

To prevent allurectl from failing the pipeline, we introduced the CL switch --silent. When using this switch, allurectl will check the server's availability, if it is not reachable, then allurectl will raise a warning in the console log, and start executing the tests as per provided CLI command without attempting to upload the test results.

Example

allurectl watch --silent -- [tests_execution_command]

Avoiding the upload of big *-result.json files which won't be processed by Allure TestOps

Feature is available starting from the allurectl release 2.15.0

Allure TestOps has limitation on the size of the *-result.json files. The size limit is set to 2,000,000 bytes (two million) bytes. If the result file exceeds the limit, then it considered as an attachment and is not processed by the parser, hence the result won't be saved to the database and won't appear in the UI. If no link to his file has been found in other *-result.json files, then it will be deleted as soon as a launch is closed.

The task

I want to avoid uploading files, which won't be processed to save time and resources for the upload and processing.

How to

We introduced CL switch --skip-too-big which will prevent uploading of the files exceeding the size of 2,000,000 bytes. The tool won't upload such files and will raise the warning in the stdout containing the information that certain files are too big to be processed by the server.

Example

allurectl watch --skip-too-big  -- [tests_execution_command]

Unload the storage by skipping attachments for successfully passed tests

The task

I want to unload the storage and hasten the results processing by avoiding the upload of attachments for successfully passed tests as these are no interest for us, we just need failed or broken tests to have the attachments.

How to #1

We introduced CL switch --ignore-passed-test-attachments in allurectl 2.9.0 which allows doing it.

This will work for the attachments in test's body. The attachments in fixtures (setup/tear-down blocks) will be uploaded even if --ignore-passed-test-attachments is specified.

Example #1

./allurectl watch --ignore-passed-test-attachments

or for the upload routine

./allurectl upload --ignore-passed-test-attachments ${ALLURE_RESULTS}

How to #2

There is ac CLI switch --exclude-files that allows you ignoring files by supplying a regular expression for files names.

The regexp to be used, has Golang flavour, so you're better off checking with something like regex101.com before applying the ignore file filers.

Example #2

This will ignore all -container.json (setup/tear-down blocks of your tests) files in allure-results directory.

./allurectl watch --exclude-files /*-container.json

or for the upload routine

./allurectl upload --exclude-files /*-container.json ${ALLURE_RESULTS}

How to read allurectl output

After the upload of test results, allurectl will render detailed information about the files that were indexed, processed and uploaded console logs.

This section tries to explain, how to read the information provided by allurectl.

Output example

Allure TestOps Client v2.15.1
Project [5] with name [eat these files]
Creating new launch with name [allurectl local upload launch 2024-01-15 113340]
Launch [34] with name [allurectl local upload launch 2024-01-15 113340]
Session [35] started
Report link: https://demo.testops.cloud/launch/34
Watcher is waiting for indexing complete...
Watcher is waiting until files processed
Total files indexed: 9229 || Finished files: 1744 || Orphan Files: 2102
...
Total files indexed: 29796 || Finished files: 28968 || Orphan Files: 828
Total files indexed: 29796 || Finished files: 28968 || Orphan Files: 828
Waiting Finished. Waited for 331ns

+---------------------------------------------------------------------------------------------------------------------------+
| Indexer Stats                                                                                                             |
+-----------+-------------------------------------------------------------+-----------------+--------------+----------------+
| INDEXING: |                         Total=29796                         | Completed=28968 |              |                |
+-----------+-------------------------------------------------------------+-----------------+--------------+----------------+
|   STATS:  |                             type                            |      count      |              |                |
+-----------+-------------------------------------------------------------+-----------------+--------------+----------------+
|           |                        allure.attach                        |      19123      |              |                |
+-----------+-------------------------------------------------------------+-----------------+--------------+----------------+
|           |                        allure.result                        |      6074       |              |                |
+-----------+-------------------------------------------------------------+-----------------+--------------+----------------+
|           |                       allure.container                      |      3610       |              |                |
+-----------+-------------------------------------------------------------+-----------------+--------------+----------------+
|           |                           unknown                           |       989       |              |                |
+-----------+-------------------------------------------------------------+-----------------+--------------+----------------+
|  TIMINGS: |                   Duration=6m26.954214603s                  |Min=325.104641ms |Avg=2m50.787s | Max=6m26.8907s |
+-----------+-------------------------------------------------------------+-----------------+--------------+----------------+
|   SKIPS:  |                            Reason                           |      Count      |              |                |
+-----------+-------------------------------------------------------------+-----------------+--------------+----------------+
|           | can't find *-result.json or *-container.json for attachment |       828       |              |                |
+-----------+-------------------------------------------------------------+-----------------+--------------+----------------+
+------------------------------------------------------------------------------------------------------------------------+
| !!! WARNING !!!                                                                                                        |
+------------------------------------------------------------------------------------------------------------+-----------+
| The following test results files have the size over 2,000,000 bytes and will be skipped by Allure TestOps: |           |
+------------------------------------------------------------------------------------------------------------+-----------+
|               /opt/allurectl/allure-results/a86295d3-483c-44e5-b1a7-ce6cdea3b1b1-result.json               | 6.751 MB  |
+------------------------------------------------------------------------------------------------------------+-----------+
|               /opt/allurectl/allure-results/24cd7f86-26e2-483c-9184-f51044debbe2-result.json               | 75.140 MB |
+------------------------------------------------------------------------------------------------------------+-----------+
+-------------------------------------------------------------------------------------------------------------------+
| Uploading Stats                                                                                                   |
+----------+--------------------------+------------------+---------------------+--------------------+---------------+
|  FILES:  | TotalBatches=3645        | TotalFiles=28968 | UploadedFiles=28968 | TotalSize=7.170 GB | ErrorsCount=0 |
+----------+--------------------------+------------------+---------------------+--------------------+---------------+
| TIMINGS: | Duration=6m25.911071126s |  Min=47.595229ms |   Avg=210.704214ms  |  Max=3.663549864s  |               |
+----------+--------------------------+------------------+---------------------+--------------------+---------------+
Watcher finished in [6m28.100628726s]
You can find report here: https://demo.testops.cloud/launch/34
Stop 2024-01-15 114008

Client version

Allure TestOps Client v2.15.1

Allure TestOps Client message shows the version of allurectl. Make sure, you are using the most recent allurectl version. This information is crucial for the troubleshooting and bugs reporting. Generally bugs must be reported only for the most recent release of allurectl.

Project and launch section

Project [5] with name [eat these files]
Creating new launch with name [allurectl local upload launch 2024-01-15 113340]
Launch [34] with name [allurectl local upload launch 2024-01-15 113340]
Session [35] started

Section reports the Id of the project where results are supposed to be uploaded to, and name of the launch to be created, as well as the result of the attempt to create the said launch. Reports the name of the created launch and session.

This information could be used for the troubleshooting purposes.

Link to created launch

Report link: https://demo.testops.cloud/launch/34

Provides the exact link to the created launch.

Index and upload process

Watcher is waiting for indexing complete...
Watcher is waiting until files processed
Total files indexed: 9229 || Finished files: 1744 || Orphan Files: 2102

When your test framework executes the tests, the allure framework integration generates the test results, these test results are indexed (allurectl detects if there are new files) and then uploaded (allurectl marks the files already uploaded in its runtime), i.e. Finished.

Orphan Files metric is being updated as the upload progresses and ideally must be zero at the end of the test run. If Orphan Files shows non-zero value, this means you have uploaded files that are not mentioned neither in -result.json nor in -container.json files. All the files that aren't mentioned in the said files.

Orphan files will be deleted on Allure TestOps side upon the launch closure event.

File type statistics

File types are thoroughly described in the documentation of Allure Report.

|   STATS:  |          type           |      count      | 
+-----------+-------------------------+-----------------+-
|           |     allure.attach       |      19123      | 
+-----------+-------------------------+-----------------+-
|           |     allure.result       |      6074       | 
+-----------+-------------------------+-----------------+-
|           |    allure.container     |      3610       | 
+-----------+-------------------------+-----------------+-
|           |        unknown          |       989       |

The output provides stats on the indexed test results.

allure.attach
- Information on the number indexed attachment files.
- *-attachment.[ext] which are explicitly named by allure framework integration naming convention as an attachment to a test result or to a fixture.
allure.result
- Information on the number of *-result.json files indexed by allurectl.
- These are the files containing the information on the tests' steps and statuses.
allure.container
- Information on the number of *-container.json files indexed by allurectl
- These are the files containing the information on the tests' fixtures steps and statuses.
unknown
- Information on the number of files indexed by allurectl that most likely are the attachments, but these are the files which names do not follow allure framework integration naming convention.
- These also could be some garbage files in terms of their usage for sake of displaying test results information in Allure TestOps.

Timings

+-----------+----------------------+---------------+--------------+----------------+
|  TIMINGS: |  Duration=6m26.95s   | Min=325.101ms | Avg=2m50.77s | Max=6m26.8992s |
+-----------+----------------------+---------------+--------------+----------------+

Here are the time characteristics of the indexing process, including the total duration, minimum, average, and maximum file indexing time.

Skips

+-----------+-------------------------------------------------------------+------------------+
|   SKIPS:  |                            Reason                           |       Count      |
+-----------+-------------------------------------------------------------+------------------+
|           | can't find *-result.json or *-container.json for attachment |        828       |
+-----------+-------------------------------------------------------------+------------------+

Section shows stats about the number of files that are not referenced in *-result.json or *-container.json files as attachments; these files will be deleted on Allure TestOps side.

Large test result files

!!! WARNING !!!
+------------------------------------------------------------------------------------------------------------+-----------+
| The following test results files have the size over 2,000,000 bytes and will be skipped by Allure TestOps: |           |
+------------------------------------------------------------------------------------------------------------+-----------+
|               /opt/allurectl/allure-results/a86295d3-483c-44e5-b1a7-ce6cdea3b1b1-result.json               | 6.751 MB  |
|               /opt/allurectl/allure-results/24cd7f86-26e2-483c-9184-f51044debbe2-result.json               | 75.140 MB |
+------------------------------------------------------------------------------------------------------------+-----------+

This section reports all test result files exceeding 2,000,000 bytes. These test results will be deleted by Allure TestOps.

Upload Statistics

+-------------------------------------------------------------------------------------------------------------------+
| Uploading Stats                                                                                                   |
+----------+--------------------------+------------------+---------------------+--------------------+---------------+
|  FILES:  | TotalBatches=3645        | TotalFiles=28968 | UploadedFiles=28968 | TotalSize=7.170 GB | ErrorsCount=0 |
+----------+--------------------------+------------------+---------------------+--------------------+---------------+
| TIMINGS: | Duration=6m25.911071126s |  Min=47.595229ms |   Avg=210.704214ms  |  Max=3.663549864s  |               |
+----------+--------------------------+------------------+---------------------+--------------------+---------------+

Upload statistics show the total number of files - including number of packages, number of uploaded files, total size and number of errors.

Batches

Batch is a group of N files, that are uploaded together from the CI pipeline to Allure TestOps.

Files are alway uploaded in batches to prevent the abuse of Allure TestOps server.

Default number (N) of files in a batch is 10.

File Information

TotalBatches

Shows number of batches uploaded to Allure TestOps.

TotalFiles

Shows total number of files generated by your tests in current pipeline.

UploadedFiles

Shows total number of files uploaded to Allure TestOps from current pipeline.

TotalSize

Shows the Total size of all files generated in the scope of current pipeline.

ErrorsCount

Shows the Number of errors during the upload process.

Duration Information

Duration
- Shows total duration of the upload process
Min
- Minimum time used to upload a single batch
Avg
- Average time used to upload a single batch
Max

Maximum time used to upload a single batch

Progress indication

Total files indexed: 29796 || Finished files: 28968 || Orphan Files: 828

When allurectl processes the files the line above is being repeated in the console with the updated information

files indexed
1. shows how many files are indexed at this very moment
Finished files
1. shows how many files are uploaded at this very moment
Orphan Files
- shows for how many files allurectl haven't found any references in the result.json or container.json files.
- this item shows what is the number of potentially useless files in current pipeline execution
- if this number is not zero at the end of allurectl execution, you probably need to check if you provided the right path to the test results
- if this number is not zero at the end of allurectl execution, you probably need to check if your tests use the files they have generated

Completion of downloading results

When the download is complete, a link to the report for the created launch and the timestamps of the end of the operation are provided.

Watcher finished in [6m28.100628726s]
You can find report here: https://demo.testops.cloud/launch/34
Stop 2024-01-15 114008

Releases

Information about allurectl releases is available on GitHub