Command line tool — allurectl

allurectl is the command line wrapper of Allure TestOps API allowing you to:

upload the test results in real time from a build job;
manage entities on Allure TestOps side (test cases, launches, projects).

Downloading allurectl

Supported platforms

To download allurectl, select the binary file suitable for your operating system and processor architecture from the available options:

MacOS on Intel x64 — **_darwin_amd64;
MacOS on M1 (ARM64) — **_darwin_arm64;
32 bits Linux — **_linux_386;
64 bits Linux — **_linux_amd64;
64 bits ARM Linux — **_linux_arm64;
32 bits Windows — **_windows_386.exe;
64 bits Windows — **_windows_amd64.exe;
64 bits ARM Windows — **_windows_arm64.exe.

The example below shows how to download allurectl for Linux on an x64 CPU:

wget https://github.com/allure-framework/allurectl/releases/latest/download/allurectl_linux_amd64 -O ./allurectl
chmod +x ./allurectl

Creating Allure TestOps API token

Before you will be able to use allurectl, you need to create Allure TestOps secret token in your user menu, so allurectl will be able to be authenticated on Allure TestOps side.

The process of getting of the API token is described in User menu article.

Working modes of allurectl

There are two modes in which allurectl operates:

Non-CI mode;
CI mode.

Non-CI mode

When allurectl starts, it checks whether the specific CI variables are defined. If they are missing, the data upload is considered local (from your PC), and allurectl runs in Non-CI mode. In this mode, you cannot use or modify parameters related to job runs.

Passing parameters to allurectl

There are 2 ways how you can send the parameters to allurectl in Non-CI mode:

using command line parameters;
using environment variables.

Upload using command line parameters

This option is convenient when you need a one-time upload for some of your tests.

Example:

allurectl upload --endpoint https://allure.company.com \
    --token 55555555-5555-5555-5555-555555555555 \
    --project-id 100 \
    --launch-name "Local PC manual launch 2020-12-31" \
    path/to/allure-results

Upload using environment variables

This option is convenient when you regularly upload your data, as it allows you to save and reuse parameters, so you don’t have to enter them in the command line each time.

Example:

# Define environment variables
export ALLURE_ENDPOINT=https://allure.company.com
export ALLURE_TOKEN=55555555-5555-5555-5555-555555555555
export ALLURE_PROJECT_ID=100
export ALLURE_LAUNCH_NAME="Local PC manual launch 2020-12-31"
export ALLURE_LAUNCH_TAGS="release, critical"
# Run upload process somewhere
allurectl upload path/to/allure-results

What will happen

If you used one of the options above:

A new launch with the name Local PC manual launch 2020-12-31 will be created on Allure TestOps side.
A new session for test results upload will be created.
Test results from the path/to/allure-results folder will be sent to the Local PC manual launch 2020-12-31 launch within the created session.
The session will be closed.
The launch will remain open until manually closed or closed base on the automatic closure rule defined for the project with ID 100.

CI mode

Each CI system has its own set of environment variables identifying it. If allurectl detects such variables, then the launch is considered as a launch from an CI system, and allurectl will operate in CI mode.

When uploading from a CI system, additional information about the system will be added to the test results, which will become available in Allure TestOps. Additionally, allurectl will use a set of internal variables to link the test results to the correct launch on the Allure TestOps side.

allurectl commands

allurectl watch

Generally, the allurectl watch command (hereafter referred to as watch) does the same things which allurectl upload (hereafter referred to as upload) does with one important difference — the watch command allows sending the test result in real time, i.e. you don't need to wait till all the tests are completed and that will decrease the workload and hasten the test results processing on Allure TestOps side.

The watch command is a wrapper for your test execution, i.e. you need to provide the command which you are using to execute your tests.

Usage

As usual you need to provide parameters needed for the connection to Allure TestOps server (in the example below these are the environment variables, and we strongly recommend you using the environment variables to pass these parameters to allurectl), then you show allurectl where you're expecting the test results to appear and then provide the command to start tests execution.

export ALLURE_ENDPOINT=https://allure.company.com
export ALLURE_TOKEN=55555555-5555-5555-5555-555555555555
export ALLURE_PROJECT_ID=100
export ALLURE_LAUNCH_NAME="Hello, world" # you can use here the environment variables of your job/pipeline
export ALLURE_LAUNCH_TAGS="release, critical"

allurectl watch --results path/to/allure-results -- ./gradlew clean test

Alternatively, you can provide all the start parameters in the environment variables, so the watch will look prettier and concise:

export ALLURE_ENDPOINT=https://allure.company.com
export ALLURE_TOKEN=542dcd56-b0e2-4fdd-8ecf-bacf0f33d505
export ALLURE_PROJECT_ID=12
export ALLURE_LAUNCH_NAME="Hello, world"
export ALLURE_RESULTS=path/to/allure-results
export ALLURE_LAUNCH_TAGS="release, critical"

allurectl watch -- ./gradlew clean test

allurectl upload

Applicability

Use allurectl with the upload command after all your tests run. We do not recommend using upload as background process.

To send data from a CI system, we recommend using the watch command. Use the upload command only if watch is not suitable for your needs.

Settings

The following environment variables could be set for easier usage of allurectl, this is an alternative to setting same parameters in the command line. Using the variables makes the execution of allurectl more clear and leaves less chance for the human error.

Environment variable	Description
`ALLURE_ENDPOINT`	URL of the Allure TestOps server
`ALLURE_PROJECT_ID`	Project ID in Allure TestOps
`ALLURE_TOKEN`	User’s personal access token created in the user menu under API Tokens
`ALLURE_LAUNCH_NAME`	Name of the launch, which will be displayed in the Allure TestOps interface
`ALLURE_LAUNCH_TAGS`	Comma-separated list of tags that will be displayed in the Allure TestOps interface

For more detailed information on configuring environment variables, refer to the sections dedicated to specific CI systems.

Tests rerun and selective run with allurectl

The most important component in rerunning and selectively running tests is the test plan. In this case, the test plan is the testplan.json file, which contains the list of test cases that your framework needs to run.

Now, let's discuss how this integration works.

Selective tests run integration

Starting from allurectl release 2.10.x, the tool and Allure TestOps fully control the process of selective test run (please mind the Allure Framework part as well), and it is not required to define ALLURE_TESTPLAN_PATH and executing job-run plan command in your pipeline. You can still use the instructions described below. However, we recommend leaving this whole process to Allure TestOps and allurectl.

On Allure TestOps side we create a list of test cases we need to (re)run on CI side. Each test case is identified by AllureID and a selector. Selector is the way test framework identifies certain test, generally it's test framework dependent.
The list of test cases is then saved to CI into testplan.json file.
On CI side we create an environment variable ALLURE_TESTPLAN_PATH and set the value to the path to testplan.json file.
When Allure Framework adaptor finds ALLURE_TESTPLAN_PATH is available, then it tries to read testplan.json.
If the file testplan.json is successfully read, then the integration instructs the test framework to run only the tests specified in the testplan.json.

If CI starts all tests from your project, then either something is not configured on your side or your test framework has no integration with Allure TestOps.

If you are facing this, please file a support (not a bug) request to our technical support.

testplan.json structure

Knowing the structure of testplan.json (see example below), you can create testplan.json on your local PC, initialize the environment variable ALLURE_TESTPLAN_PATH with path to testplan.json and run your tests locally without any additional filters in the same session. If only tests from testplan.json will run, then you have the working integration for selective run, otherwise you need to configure the integration or develop it for your test framework.

{
  "version":"1.0",
  "tests": [
    {
      "id": "10",
      "selector": "io.qameta.allure.examples.junit4.AllureSimpleTest.allureSimpleTest"
    },
    {
      "id": "11",
      "selector": "io.qameta.allure.examples.junit4.AllureParameterizedTest.allureParameterizedTest[First Name]"
    }
  ]
}

where

id is the test ID (Allure ID) from Allure TestOps;
selector is the alternative ID which is equal to test's full path by default.

Saving test plan to CI, running tests and uploading test results

In all CIs we have the same sequence:

Allure TestOps starts a job and adds ALLURE_JOB_RUN_ID = <id> to the job.

If allurectl detects ALLURE_JOB_RUN_ID then it ignores all other variables an session is created in the job run set by Allure TestOps.
Allure TestOps provides the environment variable ALLURE_TESTPLAN_PATH with path to testplan.json to the job run.
allurectl creates testplan.json and fills it with the information about the list of the tests.
allurectl executes your tests and makes the index list of the test results files and send the tests results files to Allure TestOps using watch command.
```
- `allurectl watch -- ./gradlew clean test`
```

All work with ALLURE_TESTPLAN_PATH and testplan.json is fully managed by allurectl. No user involvement and settings are required as of the allurectl release 2.10.0

Running nested jobs

It happens you need to run several jobs on a CI and send all the test results from several jobs into one aggregated launch on Allure TestOps side. For this, you can use the environment variable ALLURE_JOB_RUN_ID.

If allurectl detects that ALLURE_JOB_RUN_ID is available, then it does not create new entities but uses the one from ALLURE_JOB_RUN_ID. Same way works the execution of tests from Allure TestOps side.

Workflow

Create aggregating job on CI side.
Create new job run in the aggregating job and store the content of created ALLURE_JOB_RUN_ID.
Create nested/child job, pass the environment variable ALLURE_JOB_RUN_ID and its content to the nested/child job.

All the test results from the jobs with the same ALLURE_JOB_RUN_ID will be transferred to the same Launch in Allure TestOps.
allurectl stops the job run (no additional command is needed).

allurectl cookbook

When calling watch and upload commands, you can use --help to see the full list of supported settings. For example: allurectl watch --help.

If you have any doubts, please contact support.

Passing parameters to allurectl

Environment variables

The following environment variables need to be set for easier usage of allurectl:

Environment variable	Description
`ALLURE_ENDPOINT`	URL of the Allure TestOps server
`ALLURE_PROJECT_ID`	Project ID in Allure TestOps
`ALLURE_TOKEN`	User’s personal access token created in the user menu under API Tokens
`ALLURE_LAUNCH_NAME`	Name of the launch, which will be displayed in the Allure TestOps interface

Test connection

Please use this command only to check the connection. Do not use it in your pipelines for authorizing the test results upload process, this is done differently (see below).

This could be done from your local computer or in a CI pipeline (see remark above).

To check the connection to Allure TestOps instance, you can use the following command:

export ALLURE_TOKEN=<API-TOKEN>
export ALLURE_ENDPOINT=https://demo.testops.cloud
allurectl auth login

Alternatively, you can use CL switches, but we recommend using the environment variables.

For more information use the allurectl auth login --help command.

Upload the test results to Allure TestOps

There are two ways to download test results:

using the watch command;
using the upload command.

watch

The watch command is the recommended way to upload test results.

Generally, the watch command does the same things which upload (hereafter referred to as upload) does with one important difference — the watch command allows sending the test result in real time, i.e. you don't need to wait till all the tests are completed and that will decrease the workload and hasten the test results processing on Allure TestOps side.

The watch command is a wrapper for your test execution, i.e. you need to provide the command which you are using to execute your tests.

Usage

export ALLURE_ENDPOINT=https://allure.company.com
export ALLURE_TOKEN=55555555-5555-5555-5555-555555555555
export ALLURE_PROJECT_ID=100
export ALLURE_LAUNCH_NAME="Hello, world" # you can use here the environment variables of your job/pipeline
...
allurectl watch --results path/to/allure-results -- ./gradlew clean test

Alternatively, you can provide all the start parameters in the environment variables, so the watch will look prettier and concise:

export ALLURE_ENDPOINT=https://allure.company.com
export ALLURE_TOKEN=542dcd56-b0e2-4fdd-8ecf-bacf0f33d505
export ALLURE_PROJECT_ID=12
export ALLURE_LAUNCH_NAME="Hello, world"
export ALLURE_RESULTS=path/to/allure-results
...
allurectl watch -- ./gradlew clean test

upload

To send data from a CI system, we recommend using the watch command. Use the upload command only if watch is not suitable for your needs.

Use the upload command only when the test results are available after all tests execution (there are such test frameworks, not many but they exist) or you need to upload a directory with the test results you have.

Use allurectl with the upload command after all your tests run. We do not recommend using upload as background process.

Upload using command line parameters

allurectl upload --endpoint https://allure.company.com \
    --token 55555555-5555-5555-5555-555555555555 \
    --project-id 100 \
    --launch-name "Local PC manual launch 2200-12-31" \
    path/to/allure-results

Upload using environment variables

# Define environment variables
export ALLURE_ENDPOINT=https://demo.testops.cloud
export ALLURE_TOKEN=55555555-5555-5555-5555-555555555555
export ALLURE_PROJECT_ID=100

# Run upload process somewhere
allurectl upload --launch-name "Local PC manual launch 2200-12-31" path/to/allure-results

Getting Allure TestOps launch information

Task

We want to get the information on a launch created after the execution of watch or upload command to pass the information to chat, email, etc.

How to

This works in CI mode only.

The information on the entities created on Allure TestOps side can be placed to the environment variables and then used by invoking of the following sequence of the commands:

#define env vars
export ALLURE_ENDPOINT=https://demo.testops.cloud
export ALLURE_TOKEN=<toke>
export ALLURE_PROJECT_ID=<ID>
export ALLURE_LAUNCH_NAME="Hello, world"
export ALLURE_RESULTS=path/to/allure-results
# exec the tests
allurectl watch -- ./gradlew clean test

export $(allurectl job-run env)
# this will just show the list of all ENV vars related to allurectl execution
printenv | grep ALLURE_

printenv | grep ALLURE_ will result in the following output

ALLURE_ENDPOINT=https://demo.testops.cloud
ALLURE_LAUNCH_ID=11111
ALLURE_RESULTS=./allure-results
ALLURE_JOB_RUN_ID=12345
ALLURE_LAUNCH_TAGS=master, gitlab, demo, pytest, skip-live-doc, ignore
ALLURE_TOKEN=[MASKED]
ALLURE_LAUNCH_NAME=allure-pytest - 1ea04f48
ALLURE_JOB_RUN_URL=https://demo.testops.cloud/jobrun/14433
ALLURE_LAUNCH_URL=https://demo.testops.cloud/launch/31897
ALLURE_PROJECT_ID=433

To provide the link to the created launch, you can use either ALLURE_JOB_RUN_URL or ALLURE_LAUNCH_URL.

ALLURE_JOB_RUN_URL is an entity (there could be N job-runs) inside a launch, so if you merge two or more launches in one, then ALLURE_JOB_RUN_URL will always point to a correct launch.

Creating a launch based on test plan created in Allure TestOps project

Task

End user wants to create a test plan on Allure TestOps side, and then execute only the tests cases by referencing the test plan in the pipeline.

How to

The task can be done by creation of the testplan.json file before executing of the watch command.

export ALLURE_TOKEN=<token>
export ALLURE_ENDPOINT=https://demo.testops.cloud
export ALLURE_PROJECT_ID=111
export ALLURE_LAUNCH_NAME="$(date "+%Y-%m-%d %H%M%S") executing test plan"
export ALLURE_LAUNCH_TAGS="watch, testplan"
export ALLURE_RESULTS="allure-results"
export ALLURE_TESTPLAN_PATH="testplan.json"
# create tetsplan.json based on AQL for test cases "testPlan=222" where 222 is the ID of a test plan
./allurectl test-case plan -q "testPlan=222" --output-file ${ALLURE_TESTPLAN_PATH}
# execute test cases based on testplan.json
./allurectl watch -- [tests_execution_command]

Test result upload debugging

Task

I want to see the debug information on my upload process to troubleshoot the problems with the test results upload.

How to

Detailed debug is available starting from allurectl release 2.11.0.

allurectl has special command line switch --log which allows collecting logs of the test results upload.

./allurectl --log debug upload ${ALLURE_RESULTS}
./allurectl --log debug watch

The --log switch must be added before the watch/upload command. If added after, it will be ignored.

For each uploaded batch of files allurectl will generate line with the details on the upload process.

Logging levels

There are following logging levels from less informative to overwhelmingly informative:

fatal,
error,
warn,
info,
debug,
trace.

How to automatically add issue link to a launch

Task

I want to automatically add issue links to launches every time a launch is created from CI job.

How to

Adding issue links is available starting from allurectl release 2.2.0.

Non CI mode:

For non-CI mode, you need to know the exact ID of a launch, to attach issue information to.

allurectl launch add-issue --launch-id launch_id --integration-id integration_id --issue-keys issue_key_1,issue_key_2,...,issue_key_N

For CI mode(in pipeline after using watch command):

allurectl launch add-issue --current-launch --integration-id integration_id --issue-keys issue_key_1,issue_key_2,...,issue_key_N

where:

launch_id — launch ID;
integration_id — integration ID (you can find this ID in the Allure TestOps interface);
issue_key — task ID (for example, AE-1).

Executing tests in Docker container or on remote virtual machine / bare metal server

Task

My tests need to be executed outside the pipeline on a remote virtual machine or in a Docker container. I need the test results to be uploaded to Allure TestOps in a launch linked to the pipeline which initiated the tests execution.

How to

Feature is available starting from allurectl release 2.10.0.

allurectl allows to pack the pipeline's context and pass this information to other system, and then unpack the context in the system where your test are supposed to be executed, so allurectl in virtual machine or in a Docker container will think it's running inside a pipeline.

Pack the context

Action to be performed in the pipeline which is responsible for the triggering of the tests execution.

You need to execute allurectl with parameters ci env --encode in the scope of the parent pipeline and save the results of the command execution as ALLURE_CI_ENV environment variable.

ALLURE_CI_ENV=$(allurectl ci env --encode)

Pass the context

You need to pass the content of ALLURE_CI_ENV to the place where your tests are to be executed. This variable needs to be created and the value set before allurectl is executed.

After the variable is created and its value set, you can execute your tests by allurectl as usual.

allurectl watch -- exec-your-tests-here

The upload of test results from virtual machine or container will be linked to the pipeline which triggered the tests executions.

Executing tests even if Allure TestOps server is not reachable

Normally, if allurectl cannot connect to Allure TestOps server, it does not start the execution of the tests and fails the pipeline.

Task

I want my tests to be executed even if Allure TestOps server is not reachable du to any reason (server is in the maintenance, server is down, no route to the host, network issues, etc.).

How to

Feature is available starting from the allurectl release 2.15.0.

To prevent allurectl from failing the pipeline, we introduced the CL switch --silent. When using this switch, allurectl will check the server's availability, if it is not reachable, then allurectl will raise a warning in the console log, and start executing the tests as per provided CLI command without attempting to upload the test results.

Example

allurectl watch --silent -- [tests_execution_command]

Avoiding the upload of big *-result.json files which won't be processed by Allure TestOps

Feature is available starting from the allurectl release 2.15.0.

Allure TestOps has limitation on the size of the *-result.json files. The size limit is set to 2 000 000 bytes bytes. If the result file exceeds the limit, then it considered as an attachment and is not processed by the parser, hence the result won't be saved to the database and won't appear in the UI. If no link to his file has been found in other *-result.json files, then it will be deleted as soon as a launch is closed.

Task

I want to avoid uploading files, which won't be processed to save time and resources for the upload and processing.

How to

We introduced CL switch --skip-too-big which will prevent uploading of the files exceeding the size of 2 000 000 bytes. The tool won't upload such files and will raise the warning in the stdout containing the information that certain files are too big to be processed by the server.

Example

allurectl watch --skip-too-big  -- [tests_execution_command]

Unload storage by skipping attachments for successfully passed tests

Task

I want to unload the storage and hasten the results processing by avoiding the upload of attachments for successfully passed tests as these are no interest for us, we just need failed or broken tests to have the attachments.

How to #1

We introduced CL switch --ignore-passed-test-attachments in allurectl 2.9.0 which allows doing it.

This will work for the attachments in test's body. The attachments in fixtures (setup/tear-down blocks) will be uploaded even if --ignore-passed-test-attachments is specified.

Example #1

./allurectl watch --ignore-passed-test-attachments

or for the upload

./allurectl upload --ignore-passed-test-attachments ${ALLURE_RESULTS}

How to #2

There is ac CLI switch --exclude-files that allows you ignoring files by supplying a regular expression for files names.

The regexp to be used, has Golang flavour, so you're better off checking with something like regex101.com before applying the ignore file filers.

Example #2

This will ignore all -container.json (setup/tear-down blocks of your tests) files in allure-results directory.

./allurectl watch --exclude-files /*-container.json

or for the upload

./allurectl upload --exclude-files /*-container.json ${ALLURE_RESULTS}

How to read allurectl output

After the upload of test results, allurectl will render detailed information about the files that were indexed, processed and uploaded console logs.

This section tries to explain, how to read the information provided by allurectl.

Output example

Allure TestOps Client v2.15.1
Project [5] with name [eat these files]
Creating new launch with name [allurectl local upload launch 2024-01-15 113340]
Launch [34] with name [allurectl local upload launch 2024-01-15 113340]
Session [35] started
Report link: https://demo.testops.cloud/launch/34
Watcher is waiting for indexing complete...
Watcher is waiting until files processed
Total files indexed: 9229 || Finished files: 1744 || Orphan Files: 2102
...
Total files indexed: 29796 || Finished files: 28968 || Orphan Files: 828
Total files indexed: 29796 || Finished files: 28968 || Orphan Files: 828
Waiting Finished. Waited for 331ns

+---------------------------------------------------------------------------------------------------------------------------+
| Indexer Stats                                                                                                             |
+-----------+-------------------------------------------------------------+-----------------+--------------+----------------+
| INDEXING: |                         Total=29796                         | Completed=28968 |              |                |
+-----------+-------------------------------------------------------------+-----------------+--------------+----------------+
|   STATS:  |                             type                            |      count      |              |                |
+-----------+-------------------------------------------------------------+-----------------+--------------+----------------+
|           |                        allure.attach                        |      19123      |              |                |
+-----------+-------------------------------------------------------------+-----------------+--------------+----------------+
|           |                        allure.result                        |      6074       |              |                |
+-----------+-------------------------------------------------------------+-----------------+--------------+----------------+
|           |                       allure.container                      |      3610       |              |                |
+-----------+-------------------------------------------------------------+-----------------+--------------+----------------+
|           |                           unknown                           |       989       |              |                |
+-----------+-------------------------------------------------------------+-----------------+--------------+----------------+
|  TIMINGS: |                   Duration=6m26.954214603s                  |Min=325.104641ms |Avg=2m50.787s | Max=6m26.8907s |
+-----------+-------------------------------------------------------------+-----------------+--------------+----------------+
|   SKIPS:  |                            Reason                           |      Count      |              |                |
+-----------+-------------------------------------------------------------+-----------------+--------------+----------------+
|           | can't find *-result.json or *-container.json for attachment |       828       |              |                |
+-----------+-------------------------------------------------------------+-----------------+--------------+----------------+
+------------------------------------------------------------------------------------------------------------------------+
| !!! WARNING !!!                                                                                                        |
+------------------------------------------------------------------------------------------------------------+-----------+
| The following test results files have the size over 2,000,000 bytes and will be skipped by Allure TestOps: |           |
+------------------------------------------------------------------------------------------------------------+-----------+
|               /opt/allurectl/allure-results/a86295d3-483c-44e5-b1a7-ce6cdea3b1b1-result.json               | 6.751 MB  |
+------------------------------------------------------------------------------------------------------------+-----------+
|               /opt/allurectl/allure-results/24cd7f86-26e2-483c-9184-f51044debbe2-result.json               | 75.140 MB |
+------------------------------------------------------------------------------------------------------------+-----------+
+-------------------------------------------------------------------------------------------------------------------+
| Uploading Stats                                                                                                   |
+----------+--------------------------+------------------+---------------------+--------------------+---------------+
|  FILES:  | TotalBatches=3645        | TotalFiles=28968 | UploadedFiles=28968 | TotalSize=7.170 GB | ErrorsCount=0 |
+----------+--------------------------+------------------+---------------------+--------------------+---------------+
| TIMINGS: | Duration=6m25.911071126s |  Min=47.595229ms |   Avg=210.704214ms  |  Max=3.663549864s  |               |
+----------+--------------------------+------------------+---------------------+--------------------+---------------+
Watcher finished in [6m28.100628726s]
You can find report here: https://demo.testops.cloud/launch/34
Stop 2024-01-15 114008

Client version

Allure TestOps Client v2.15.1

Allure TestOps Client message shows the version of allurectl. Make sure, you are using the most recent allurectl version. This information is crucial for the troubleshooting and bugs reporting. Generally bugs must be reported only for the most recent release of allurectl.

Project and launch section

Project [5] with name [eat these files]
Creating new launch with name [allurectl local upload launch 2024-01-15 113340]
Launch [34] with name [allurectl local upload launch 2024-01-15 113340]
Session [35] started

Section reports the project ID where results are supposed to be uploaded to, and the name of the launch to be created, as well as the result of the attempt to create the said launch. Reports the name of the created launch and session.

This information could be used for the troubleshooting purposes.

Link to created launch

Report link: https://demo.testops.cloud/launch/34

Provides the exact link to the created launch.

Index and upload process

Watcher is waiting for indexing complete...
Watcher is waiting until files processed
Total files indexed: 9229 || Finished files: 1744 || Orphan Files: 2102

When your test framework executes the tests, the allure framework integration generates the test results, these test results are indexed (allurectl detects if there are new files) and then uploaded (allurectl marks the files already uploaded in its runtime), i.e. Finished.

Orphan Files metric is being updated as the upload progresses and ideally must be zero at the end of the test run. If Orphan Files shows non-zero value, this means you have uploaded files that are not mentioned neither in -result.json nor in -container.json files. All the files that aren't mentioned in the said files.

Orphan files will be deleted on Allure TestOps side upon the launch closure event.

File type statistics

File types are thoroughly described in the documentation of Allure Report.

|   STATS:  |          type           |      count      | 
+-----------+-------------------------+-----------------+-
|           |     allure.attach       |      19123      | 
+-----------+-------------------------+-----------------+-
|           |     allure.result       |      6074       | 
+-----------+-------------------------+-----------------+-
|           |    allure.container     |      3610       | 
+-----------+-------------------------+-----------------+-
|           |        unknown          |       989       |

The output provides stats on the indexed test results.

allure.attach
- Information on the number indexed attachment files.
- *-attachment.[ext] which are explicitly named by allure framework integration naming convention as an attachment to a test result or to a fixture.
allure.result
- Information on the number of *-result.json files indexed by allurectl.
- These are the files containing the information on the tests' steps and statuses.
allure.container
- Information on the number of *-container.json files indexed by allurectl
- These are the files containing the information on the tests' fixtures steps and statuses.
unknown
- Information on the number of files indexed by allurectl that most likely are the attachments, but these are the files which names do not follow allure framework integration naming convention.
- These also could be some garbage files in terms of their usage for sake of displaying test results information in Allure TestOps.

Timings

+-----------+----------------------+---------------+--------------+----------------+
|  TIMINGS: |  Duration=6m26.95s   | Min=325.101ms | Avg=2m50.77s | Max=6m26.8992s |
+-----------+----------------------+---------------+--------------+----------------+

Here are the time characteristics of the indexing process, including the total duration, minimum, average, and maximum file indexing time.

Skips

+-----------+-------------------------------------------------------------+------------------+
|   SKIPS:  |                            Reason                           |       Count      |
+-----------+-------------------------------------------------------------+------------------+
|           | can't find *-result.json or *-container.json for attachment |        828       |
+-----------+-------------------------------------------------------------+------------------+

Section shows stats about the number of files that are not referenced in *-result.json or *-container.json files as attachments; these files will be deleted on Allure TestOps side.

Large test result files

!!! WARNING !!!
+------------------------------------------------------------------------------------------------------------+-----------+
| The following test results files have the size over 2,000,000 bytes and will be skipped by Allure TestOps: |           |
+------------------------------------------------------------------------------------------------------------+-----------+
|               /opt/allurectl/allure-results/a86295d3-483c-44e5-b1a7-ce6cdea3b1b1-result.json               | 6.751 MB  |
|               /opt/allurectl/allure-results/24cd7f86-26e2-483c-9184-f51044debbe2-result.json               | 75.140 MB |
+------------------------------------------------------------------------------------------------------------+-----------+

This section reports all test result files exceeding 2 000 000 bytes. These test results will be deleted by Allure TestOps.

Upload Statistics

+-------------------------------------------------------------------------------------------------------------------+
| Uploading Stats                                                                                                   |
+----------+--------------------------+------------------+---------------------+--------------------+---------------+
|  FILES:  | TotalBatches=3645        | TotalFiles=28968 | UploadedFiles=28968 | TotalSize=7.170 GB | ErrorsCount=0 |
+----------+--------------------------+------------------+---------------------+--------------------+---------------+
| TIMINGS: | Duration=6m25.911071126s |  Min=47.595229ms |   Avg=210.704214ms  |  Max=3.663549864s  |               |
+----------+--------------------------+------------------+---------------------+--------------------+---------------+

Upload statistics show the total number of files - including number of packages, number of uploaded files, total size and number of errors.

Batches

Batch is a group of N files, that are uploaded together from the CI pipeline to Allure TestOps.

Files are alway uploaded in batches to prevent the abuse of Allure TestOps server.

Default number (N) of files in a batch is 10.

File Information

TotalBatches Shows number of batches uploaded to Allure TestOps.
TotalFiles Shows total number of files generated by your tests in current pipeline.
UploadedFiles Shows total number of files uploaded to Allure TestOps from current pipeline.
TotalSize Shows the Total size of all files generated in the scope of current pipeline.
ErrorsCount Shows the Number of errors during the upload process.

Duration Information

Duration Shows total duration of the upload process.
Min Minimum time used to upload a single batch.
Avg Average time used to upload a single batch.
Max Maximum time used to upload a single batch.

Progress indication

Total files indexed: 29796 || Finished files: 28968 || Orphan Files: 828

When allurectl processes the files the line above is being repeated in the console with the updated information.

Total files indexed Shows how many files are indexed at this very moment.
Finished files Shows how many files are uploaded at this very moment.
Orphan Files
- Shows for how many files allurectl haven't found any references in the result.json or container.json files.
- This item shows what is the number of potentially useless files in current pipeline execution.
- If this number is not zero at the end of allurectl execution, you probably need to check if you provided the right path to the test results.
- If this number is not zero at the end of allurectl execution, you probably need to check if your tests use the files they have generated.

Completion of downloading results

When the download is complete, a link to the report for the created launch and the timestamps of the end of the operation are provided.

Watcher finished in [6m28.100628726s]
You can find report here: https://demo.testops.cloud/launch/34
Stop 2024-01-15 114008

Releases

Information about allurectl releases is available on GitHub