Test results

In a launch, a test result represents one attempt of executing a test. It is created either manually or automatically by uploading a test result file.

• In the most basic situation, a test result can be thought of as a simple record containing the date and time of the attempt and its status, e.g., “Passed” or “Failed”.

• For a manual test with a scenario, a QA engineer sets individual statuses for all its steps, with optional attachments. For the test result as a whole, they can also provide issue links, tags, and custom fields describing the attempt.

  An automated test result may include such data, too. Check out Allure Report documentation to learn more about steps, attachments and metadata that can be provided when running a test.

• Each test result can also describe its environment — for example, the operating system and browser the QA engineer used. Depending on the configuration and how the launch was created, the environments may or may not be identical for all test results in a launch. To learn more about this feature, see Environment.

When you close the launch, all test results in it get processed. Allure TestOps then uses the test results to create or update the test cases and all related analytics in the project.

Test statuses

Each completed test result has a test status. These statuses are reflected in various statistics and graphs throughout the Allure TestOps interface, and they also can be used for searching and filtering test cases.

When running a test manually, the newly created (incomplete) test result get the temporary In progress pseudo-status initially, which is then changed to another status by a QA engineer. For automated test results, the status is loaded from a test result file (see also the Test statuses article in the Allure Report documentation).

There are five supported statuses.

• Passed (green): the test finished successfully.
• Failed (red): the test encountered an unexpected behavior in the system under test. This means that the test itself seems to be valid (not Broken), but its execution ended with a false assertion.
• Skipped (gray): the test was included in the test plan but then not executed for some reason.
• Broken (orange): the test failed because of a test defect. Unlike Failed, this status means that the test was unable to check the product's behavior as it intended to, therefore, the failure may or may not indicate an actual product defect.
• Unknown (violet): the test status was not explicitly reported. In a manual test, this means that the launch was closed while the test was still In progress. In an automated test, this is most likely due to a bug in the Allure Report adapter that was used.

If you use test case scenarios, you can also set individual statuses for each step and sub-step. Step statuses do not directly affect the test statuses and may be used for different purposed by a QA engineer.

Supported file formats

Allure TestOps recognizes various data formats of automated test results. Files in any of these formats can be uploaded via the web interface or via allurectl.

• Allure Report 2.x format — *-result.json, *-container.json, *-attachment.*
• Allure Report 1.x format — *.xml (supported for compatibility with older adapters and integrations)
• Cucumber reporting format — *.json
• JGiven reporting format — *.json
• JUnit XML reporting format — *.xml
• Visual Studio reporting format — *.trx (very old, not recommended to use)
• XCTest reporting format — *.xcresult (only files from XCode 11 and higher are supported)
• xUnit.net reporting format — *.xml

Matching test results to test cases

When you upload test results to Allure TestOps, for each uploaded result Allure TestOps does one of the following:

• Creates a new test case if this is the first uploaded result for that test case.
• Updates an existing test case if you have already uploaded results for it before.
• Marks the uploaded result as orphaned if there is invalid data (see below for more info).

After the initial upload, all subsequent uploads need to be matched with the created test case. To do this, Allure adapters generate a unique ID for each test case, separate from the ID you see in the Allure TestOps interface. This ID is a hash of the data provided by the testing framework.

If the testing framework does not provide a ready-made test case identifier, the adapter uses the signature of a test function combined with additional data like package name to make it more unique.

This process can be represented as follows:

testCaseId = md5(fullName + sort(params))

where fullName is the name of the function with some extra information (e.g., my_package#my_function), and params are the function parameters.

Then, after the upload, Allure TestOps will try to find an existing test case with that ID and update it with the new data. If no such test case exists, it will create a new one.

If you don't use an Allure adapter and your test results don't have this field, Allure TestOps will generate this ID automatically when you upload the test results. It won't necessarily be the same as the value generated by an adapter.

A potential issue with identifiers generated this way is their sensitivity to code changes. For example, if a function's signature was used to create the ID and you later modify the function name or its parameters, the identifier will change. This results in Allure TestOps creating a new test case instead of updating the existing one.

To avoid this, after the first test result upload, you can modify your code to explicitly specify the test case ID from Allure TestOps:

You cannot specify an ID that does not yet exist in Allure TestOps. If you do that, the uploaded test results will be orphaned and no test cases will be created or updated.

@Test
@AllureId("123")
public void myTest() {
   step("Step 1", () -> {});
}

it("My Test @allure.id:123", async () => {
   await allure.step("Step 1", async () => {});
});

@allure.id("123")
def my_test():
   with allure.step("Step 1"):
      assert 1 == 1

After you run the test, the adapter will create a test result file with the following fields:

"testCaseId": "db57b9d129d73c4b4d9a1d1023774f53",
"labels": [
   {
      "name": "as_id",
      "value": "123"
   }
]

Here, testCaseId is a test case identifier generated by the adapter (a hash of data provided by the testing framework). In the example above, instead of testCaseId, the explicitly specified test case ID from Allure TestOps will be used — as_id (or allure_id in some adapters).