Test results

In a launch, a test result represents one attempt of executing a test. It is created either manually or automatically by uploading a test result file.

At the most basic level, a test result answers these questions:

• what test was executed;
• when and where it ran;
• what status it finished with;
• what evidence was attached to that attempt;
• how that attempt relates to previous history, retries, defects, and issues.

For a manual test with a scenario, a QA engineer sets individual statuses for all test steps, optionally including attachments. For the test result as a whole, they can also provide issue links, tags, and custom fields.

An automated test result may include such data, too. Check out Allure Report documentation to learn more about steps, attachments and metadata that can be provided when running a test.

Each test result can also describe its environment, for example the operating system and browser used during the run. Depending on the configuration and how the launch was created, the environments may or may not be identical for all the test results in a launch. To learn more about this feature, see Environment.

When you close a launch, Allure TestOps processes all test results the launch contains. These test results are then used to create or update the relevant test cases and analytical data in the project.

Test result page anatomy

The detailed result page is where you inspect one execution attempt deeply.

Main tabs

Each test result exposes four main tabs:

• Overview — the main report for that single result.
• History — earlier results of the same logical test across launches.
• Retries — repeated attempts of the same execution context.
• Attachments — files and artifacts collected during the execution.

Overview tab

The overview tab is the main inspection surface. Depending on the test and project settings, it may include:

• status details;
• category;
• defect links;
• similar failures;
• parameters;
• description;
• links;
• execution or scenario details;
• comments;
• optional sections such as precondition or expected result.

Metadata sidebar

The metadata sidebar gives the context needed for triage. It can include:

• layer;
• stop time and duration;
• history preview;
• retries preview;
• named environment;
• tags;
• issues;
• test key;
• members;
• environment variables;
• custom fields.

Actions on an open launch

When the parent launch is still open, the result page can expose actions such as:

• assign;
• link defect;
• mute or unmute;
• rerun;
• resolve an in-progress manual result.

These actions are part of the active review workflow. Once the launch is closed, the result becomes part of durable reporting instead of an editable working queue.

Test statuses

Each completed test result has a test status. These statuses are reflected in various statistics and graphs throughout the Allure TestOps interface, and they also can be used for searching and filtering test cases.

When running a test manually, the newly created (incomplete) test result initially gets a temporary In progress pseudo-status, which is then changed to another status by a QA engineer. For automated test results, the status is loaded from a test result file (see also the Test statuses article in the Allure Report documentation).

There are five test statuses in Allure TestOps:

• Passed (green) — test finished successfully.
• Failed (red) — unexpected behavior was encountered during the test run. This means that the test itself seems to be valid (not Broken), but the test run ended with an invalid behavior (e.g., false assertion).
• Skipped (gray) — test is part of a test plan but was skipped for some reason.
• Broken (orange) — test run was interrupted because of a test defect. Unlike Failed, this status means that the test was unable to check the product behavior as intended, therefore this status may or may not indicate an actual problem with the product.
• Unknown (violet) — test status was not explicitly reported. In a manual test, this means that the launch was closed while the test was still In progress. In an automated test, this is most likely due to a bug in the Allure Report adapter that was used.

In manual test case scenarios, you can set individual statuses for each step and sub-step. These statuses do not directly affect the overall status of a test and may be used for different purposes by QA engineers.

These five statuses are the fixed execution-status model used throughout TestOps analytics. Teams can enrich triage with categories, defects, mute state, and workflow metadata, but they do not add new base execution statuses.

Execution status vs triage resolution

Execution status tells you how the run ended. Resolution tells you whether the team has already triaged that failure.

A failed result is considered resolved once a person has applied an explicit decision such as:

• linking the result to a defect;
• muting the result because the failure is understood and intentionally suppressed.

If neither of those has happened yet, the failure remains unresolved and continues to appear in unresolved-review surfaces such as launch overview and defect triage workflows.

Sorting and filtering test results

When viewing a launch, you can sort and filter the list of test results in the same way as the list of test cases.

To sort a list of test results:

1. Click Display.
2. From the Sort by menu, select the field by which you want to sort the test results.
3. From the Direction menu, select the direction of sorting.

To filter a list of test results, click the search box above the list and select the field and value required.

For example, to show only automated tests in the list:

1. Click the search box.
2. Select Type in the drop-down list.
3. Select Automated.

You can find more information on working with filters in the Test cases article.

Bulk actions

If a launch consists of a large number of tests, you can use the bulk actions menu to modify several test results at once:

The maximum number of test results that can be selected for bulk actions:

• in the cloud version of Allure TestOps — 10 000;
• in the server version of Allure TestOps — 50 000.

1. Select the test results using Ctrl or by selecting the checkboxes next to them.
2. Click Status in the bottom panel to change the status, or click ⋯ to select a different action.

Using the bulk actions menu you can:

• change test result statuses;
• assign team members;
• rerun tests;
• link tests to defects;
• mute test cases;
• add and remove tags;
• export test results to CSV;
• remove test results from a launch.

If a launch is closed, only the Export CSV action will be available in the menu.

History vs retries

The History and Retries tabs solve different questions.

History

Use History when you want to understand how the same logical test behaved across different launches over time.

Typical questions for the history tab are:

• "Has this test been failing for several launches already?"
• "Did it start failing only in the latest release?"
• "What changed compared with previous runs of the same test?"

History is cross-launch context.

Retries

Use Retries when you want to inspect repeated attempts within the same execution context.

Typical questions for the retries tab are:

• "Did this test pass after a rerun?"
• "Is the current failure unstable or reproducible?"
• "What did the previous attempt inside this launch look like?"

Retries are intra-run context.

Reruns and hidden retry results

When you rerun tests from an open launch, Allure TestOps keeps the new result in the same launch instead of forcing you to compare several separate launches.

In that situation:

• the latest attempt becomes the current visible result for normal review;
• earlier attempts remain available in Retries and History;
• older retry attempts may be hidden from the main tree to reduce noise while still remaining part of the audit trail.

This is why a rerun can improve the launch-level signal without erasing the fact that earlier attempts existed.

Retries also depend on execution context stability. If parameters or mapped environment values change between attempts, the incoming result may stop matching the expected retry and instead appear as a separate result.

Supported file formats

Allure TestOps supports several file formats for importing results of automated tests. You can upload these files via the web interface or using allurectl.

• Allure Report 2.x format — *-result.json, *-container.json, *-attachment.*;
• Allure Report 1.x format — *.xml (supported for compatibility with older adapters and integrations);
• Cucumber reporting format — *.json;
• JGiven reporting format — *.json;
• JUnit XML reporting format — *.xml;
• Visual Studio reporting format — *.trx (obsolete, not recommended to use);
• XCTest reporting format — *.xcresult (only files from XCode 11 and higher are supported);
• xUnit.net reporting format — *.xml.

Matching test results to test cases

Three different identity layers are involved in automated-result processing:

• Allure ID — the explicit stable identity of a test case in TestOps.
• testCaseId — the adapter-generated identity used to match future uploads to the same logical automated test case.
• historyId — the execution-context identity used to relate expected and actual attempts, especially for retries and reruns.

When you upload test results to Allure TestOps, for each uploaded result Allure TestOps does one of the following:

• Creates a new test case if this is the first uploaded result for that test case.
• Updates an existing test case if you have already uploaded results for it before.
• Marks the uploaded result as orphaned if there is invalid data (see below for more info).

After the initial upload, all subsequent uploads need to be matched with the created test case. To do this, Allure adapters generate a unique ID for each test case, separate from the ID you see in the Allure TestOps interface. This ID is a hash of the data provided by the testing framework.

If the testing framework does not provide a ready-made test case identifier, the adapter uses the signature of a test function combined with additional data like package name to make it more unique.

This process can be represented as follows:

testCaseId = md5(fullName + sort(params))

where fullName is the name of the function with some extra information (e.g., my_package#my_function), and params are the function parameters.

Then, after the upload, Allure TestOps will try to find an existing test case with that ID and update it with the new data. If no such test case exists, it will create a new one.

If you don't use an Allure adapter and your test results don't have this field, Allure TestOps will generate this ID automatically when you upload the test results. It won't necessarily be the same as the value generated by an adapter.

A potential issue with identifiers generated this way is their sensitivity to code changes. For example, if a function's signature was used to create the ID and you later modify the function name or its parameters, the identifier will change. This results in Allure TestOps creating a new test case instead of updating the existing one.

To avoid this, after the first test result upload, you can modify your code to explicitly specify the test case ID from Allure TestOps:

You cannot specify an ID that does not yet exist in Allure TestOps. If you do that, the uploaded test results will be orphaned and no test cases will be created or updated.

@Test
@AllureId("123")
public void myTest() {
   step("Step 1", () -> {});
}

it("My Test @allure.id:123", async () => {
   await allure.step("Step 1", async () => {});
});

@allure.id("123")
def my_test():
   with allure.step("Step 1"):
      assert 1 == 1

After you run the test, the adapter will create a test result file with the following fields:

"testCaseId": "db57b9d129d73c4b4d9a1d1023774f53",
"labels": [
   {
      "name": "as_id",
      "value": "123"
   }
]

Here, testCaseId is a test case identifier generated by the adapter (a hash of data provided by the testing framework). In the example above, instead of testCaseId, the explicitly specified test case ID from Allure TestOps will be used — as_id (or allure_id in some adapters).

Why reruns and retries can split unexpectedly

historyId depends on the execution context, not just on the logical test case. In practice this means a rerun can split into a separate result when:

• parameter values change between attempts;
• the set of parameters changes;
• mapped environment values change between attempts.

When that happens, TestOps no longer sees the incoming result as the same retry context. The expected result may stay In progress until the launch is closed, while the new incoming result appears separately.

Attachments and fixture artifacts

Attachments are a great way to enrich the test result so that a QA engineer performing triage can understand what happened during the test execution. When using Allure Framework, you can attach different types of files to your test results, and Allure TestOps will process these the same way as Allure Report does.

The Attachments tab separates:

• test-body attachments;
• fixture attachments.

This distinction is useful when the failure may have happened during setup or teardown rather than during the main test body.

Many attachment types can be previewed inline, which makes the result page a practical investigation surface instead of only a download list.

When to move from result review to another tool

After inspecting a test result, the next step usually belongs to one of these workflows:

• Defects when the result matches or creates a known-failure pattern.
• Compare launches when the real question is what changed across runs.
• Dashboards when the question is broader than one result and needs project-level trends.
• Test cases when you need to inspect or edit the underlying test asset itself.

Next steps

• Defects for triage and known-failure handling.
• Launches for the parent execution context.
• Compare launches for cross-launch analysis.