Environment

Sometimes a test's outcome can be affected by the environment in which it is executed. Allure TestOps provides a way to describe each test's environment (in the form of key-value pairs) and makes sure that test results from different environments are not merged together.

The environment becomes a part of a test result and can be used when filtering test results via the web interface or the query language.

Examples

• When working with a web application, a manual QA engineer may want to run the same test case in different web browsers. To do so, they define multiple environments when creating a launch, each one with a different value for “Browser”. Allure TestOps generates one test result for each value, allowing the QA engineer to report their outcomes independently.

• In a complex CI/CD setup, automated tests may be running on different nodes or even different operating systems. By mapping environment variables to Allure TestOps environment, the QA team can make the different runs produce separate results (not just retries). This can be useful for future analysis of the test's performance on different platforms, etc.

Configuration

1. Declare environment variables

Before you can start assigning a new environment variable to your tests (either manual or automated), an instance administrator must declare the variable globally in Administration → Environment. This is to help you maintain the same naming conventions across different projects.

2. Create automated mappings

This step affects automated testing. If you only use environments for manual tests, you can skip this step.

The CI integration plugins and the allurectl utility use the operating system's environment variables as the mechanism for defining the environment for the test runtime. It is assumed that the test implementation will read relevant variables and change its behavior accordingly: for example, run the specified browser instead of the default one.

To set up this mechanism, a project owner must map the human-readable names (declared on the previous step) to the actual names of variables that the test code will look for. This is done in Project settings → Environment.

The mapping works in both directions.

• When you initiate a job from Allure TestOps, it passes the necessary values to the CI server, where an integration plugin adds them to the test runtime's environment.

• When a plugin or the utility sends test results back to Allure TestOps, they attach the full list of environment variables from the runtime. If the list contains any names for which a mapping exists, Allure TestOps will display their values in the test result.

Usage

Run tests in environments

By default, when you select a test and choose Run, Allure TestOps runs it once. The default environment for the test is either an empty environment (for manual test cases) or the selected job's default environment (for automated test cases). However, it is possible to define one or even multiple environments in the Run dialog, thus creating a separate test result for each environment.

Note that the Environment section of the dialog works as a two-level list: first you add an environment, then you add a variable to the environment. For example, if you need an environment with multiple variables, make sure to click the inner Add button, not the outer one.

The screenshot below demonstrates a configuration that will run each test 3 times (for the automated tests, this means triggering 3 job runs). When used on 5 test cases, such a configuration will create a launch with 15 different test results.

Upload test results with environments

When you upload automated test results to a new or existing launch, you can set the environment variables in the Environment section of the dialog.

These values will be set for all test results that you upload in the current dialog, which are not necessarily all test results in a launch. For example, you may run your tests on multiple machines and individually upload their results into the same launch, each time with a different value for “Host” or some other environment variable.

Set the default variables for a job

When you configure a job, you can specify the default environment the job will use. Each variable not only has a default value, but also is mapped to an environment variable name from the CI server. When the job is triggered from the CI server, this variable can be used to override the default value.

For launches triggered from Allure TestOps (by clicking Run on a test case, a test plan, or the job itself), overriding the default value is possible, too.

• If a launch has one environment, its variables override the job's defaults for every task.

• If a launch has two or more environments, Allure TestOps will create the corresponding amount of job runs, and in each job run the defaults will be overridden by a different environment.

Note that some integrations may need to have a special variable called Branch in every job's default environment. Check the instructions in the CI integrations section.