Test results
In a launch, a test result represents one attempt of executing a test. It is created either manually or automatically by uploading a test result file.
In the most basic situation, a test result can be thought of as a simple record containing the date and time of the attempt and its status, e.g., “Passed” or “Failed”.
For a manual test with a scenario, the tester sets individual statuses for all its steps, with optional attachments. For the test result as a whole, they can also provide issue links, tags, and custom fields describing the attempt.
An automated test result may include such data, too. Check out Allure Report documentation to learn more about steps, attachments and metadata that can be provided when running a test.
Each test result can also describe its environment — for example, the operating system and browser the tester used. Depending on the configuration and how the launch was created, the environments may or may not be identical for all test results in a launch. To learn more about this feature, see Environment.
When you close the launch, all test results in it get processed. Allure TestOps then uses the test results to create or update the test cases and all related analytics in the project.
Test statuses
Each completed test result has a test status. These statuses are reflected in various statistics and graphs throughout the Allure TestOps interface, and they also can be used for searching and filtering test cases.
When running a test manually, the newly created (incomplete) test result get the temporary In progress pseudo-status initially, which is then changed to another status by the tester. For automated test results, the status is loaded from a test result file (see also the Test statuses article in the Allure Report documentation).
There are five supported statuses.
- Passed (green): the test finished successfully.
- Failed (red): the test encountered an unexpected behavior in the system under test. This means that the test itself seems to be valid (not Broken), but its execution ended with a false assertion.
- Skipped (gray): the test was included in the test plan but then not executed for some reason.
- Broken (orange): the test failed because of a test defect. Unlike Failed, this status means that the test was unable to check the product's behavior as it intended to, therefore, the failure may or may not indicate an actual product defect.
- Unknown (violet): the test status was not explicitly reported. In a manual test, this means that the launch was closed while the test was still In progress. In an automated test, this is most likely due to a bug in the Allure Report adapter that was used.
If you use test case scenarios, you can also set individual statuses for each step and sub-step. Step statuses do not directly affect the test statuses and may be used for different purposed by the tester.
Supported file formats
Allure TestOps recognizes various data formats of automated test results. Files in any of these formats can be uploaded via the web interface or via allurectl.
- Allure Report 2.x format —
*-result.json,*-container.json,*-attachment.* - Allure Report 1.x format —
*.xml(supported for compatibility with older adapters and integrations) - Cucumber reporting format —
*.json - JGiven reporting format —
*.json - JUnit XML reporting format —
*.xml - Visual Studio reporting format —
*.trx(very old, not recommended to use) - XCTest reporting format —
*.xcresult(only files from XCode 11 and higher are supported) - xUnit.net reporting format —
*.xml
