TestOps logo
Try TestOps

Run automated tests

Automated execution starts from code and ends in Allure TestOps as launches, test results, and eventually test cases.

Automated execution is where project structure, CI integration, environment mapping, and result ingestion meet. The core model is simple:

  1. your framework runs the tests and produces result files;
  2. those files are uploaded to Allure TestOps;
  3. Allure TestOps stores them in a launch;
  4. after the launch is closed, Allure TestOps updates the visible automated test base and analytics.

What happens in Allure TestOps

An automated execution normally goes through these stages:

  1. Your framework runs the tests and produces result files.
  2. Those files are uploaded to Allure TestOps, either manually, from CI, or through a run started from Allure TestOps itself.
  3. Allure TestOps creates or updates a launch for that execution.
  4. If the run came from a connected CI job, the automated part of that launch is grouped into one or more job runs.
  5. While the launch is open, you can continue uploading, reviewing, rerunning, and triaging.
  6. When you close the launch, Allure TestOps updates the corresponding automated test cases and refreshes analytics.

Choose an execution path

There are three main ways to work with automated tests in Allure TestOps.

Manual upload

Use manual upload when you need to:

  • verify that result files are generated correctly;
  • test a new adapter or pipeline step;
  • import one-off execution data without changing CI.

In this model:

  • you run the tests outside Allure TestOps;
  • you upload the produced files in Launches;
  • Allure TestOps creates a launch and stores the results there.

See Running an automated test case for the step-by-step evaluation flow.

Automatic upload from CI or allurectl

Use this as the normal day-to-day workflow once your team wants repeatable ingestion.

  • A CI plugin or allurectl uploads results automatically.
  • The upload creates a launch and, when applicable, a job run linked to the CI execution.
  • The launch becomes the shared execution record for review, reruns, defect matching, and comparison.

This is the best default when your team already has a working pipeline and wants launches to appear as part of that pipeline.

Plugin-based CI integrations such as Jenkins, TeamCity, and Bamboo own the upload inside the CI platform. GitHub Actions, GitLab CI, AWS CodePipeline, and similar pipelines usually use allurectl. Both models are valid as long as the pipeline and TestOps agree on the same launch and job-run context.

Execution started from Allure TestOps

Use this path when you want Allure TestOps to define the scope first and let CI execute only that scope.

  • You select test cases directly, run a test plan, or start a job.
  • Allure TestOps creates the launch before the pipeline starts.
  • The selected job receives the execution context and uploads the results back into that launch.

This is the path used for selective execution, planned release runs, and mixed manual-plus-automation workflows.

Launches, jobs, and job runs

These three terms matter in automated workflows:

  • Launch: the shared execution context that contains the test results for one testing activity.
  • Job: the reusable link between an Allure TestOps project and a CI pipeline definition.
  • Job run: one execution of that job inside a launch.

One launch may contain multiple job runs. This is common when:

  • a launch is started from a connected job;
  • the same launch contains several environment variants for the automated scope;
  • an automated rerun is triggered from the launch;
  • several pipelines are deliberately consolidated into one launch.

Manual test results belong to the launch as well, but they are not grouped into job runs.

Selective execution prerequisites

Selective execution means that Allure TestOps decides which tests should run and the test framework executes only that subset.

Use selective execution when you want to:

  • run a specific test plan;
  • run selected automated test cases from the tree;
  • rerun only the tests that matter for the current task.

For this to work correctly:

  1. Your automated tests must use an Allure adapter that provides enough identity data for test selection.
  2. The framework integration must support test-plan-based selective execution.
  3. The job must be configured so it can be started from Allure TestOps.
  4. The automated test cases in Allure TestOps must already be linked to the correct job or jobs through earlier uploads and executor assignment.

If the framework or adapter does not honor the generated test plan, Allure TestOps can still start the job, but the CI system may execute the whole suite instead of only the selected subset.

For the TestOps-to-CI selective run contract, see allurectl.

Rerun prerequisites

Reruns are a special case of selective automated execution. They reuse the current launch context and create a new job run inside the same launch.

Reruns work best when:

  • the launch came from a connected CI job or from a launch that already knows which job should execute the test again;
  • the framework integration supports test-plan-based selection;
  • the current launch stays open while the team is still reviewing and verifying results.

If a rerun cannot be scoped precisely, the platform may need to re-execute a broader pipeline scope than the selected test results suggest.

Keeping the launch open matters because the rerun needs a live execution context. If the original launch is already finalized, the next upload may still arrive successfully but it will no longer behave like an in-launch rerun.

Environments and launch metadata

Automated runs are more useful when the launch clearly states which release, branch, browser, platform, or host produced the results.

The most common places to control that metadata are:

  • the launch creation dialog when you start a run from Allure TestOps;
  • environment mappings at project level;
  • job defaults in Jobs;
  • CI or allurectl variables that are mapped into launch environments and other launch metadata.

Environment values affect result grouping and filtering. They also help make sure runs from materially different runtime contexts are not interpreted as the same execution.

Metadata ownership for automated test cases

Automated test cases can receive metadata from two places:

  • uploaded test results;
  • the test case stored in Allure TestOps.

The default model for automation-heavy teams is to keep most metadata code-driven and let uploads update the test case automatically. Use Metadata update settings only when some fields must stay owned in the UI, for example a manually maintained description or expected result.

Where troubleshooting starts

If the run does not behave as expected, start from the surface that owns the problem:

Next steps

© Qameta Software Inc. All rights reserved.