Test results FAQ
This page covers possible causes of the situations when there are missing test results or there is inconsistency between different views in Allure TestOps, e.g when the number of test results is less in overview and correct in the Tree section of a launch and the some others.
Covenants
To make some phrases easier and shorter for reading we'll use Allure TestOps agent or simply – agent to refer to Allure TestOps plug-in and/or allurectl command line tool.
Missing results - Empty launch - Wrong or empty test results source directory
Symptom: This could be a case when your launch is totally empty. You need to check if you are referring to correct test results directory.
This is to be checked in the settings of Allure TestOps agent used to upload the test results.
You also can execute ls -a command during the build for the path you used for settings of Allure TestOps agent if there are any test results in this directory. If there isn't any, then you need to check for the right paths.
How to fix
The only way to fix this is to point to the right directory with test results.
Check what is the right path where your tests save the results, then update settings for Allure TestOps agent.
Missing results - Empty launch - Files exist on CI, paths are correct but files aren't uploaded
If you can see the following log details in a plugin or in logs of allurectl:
Launch [LL], job run [rr], session [ss]:files ignored [0], indexed [XX], processed [0], errors [XX]
or
[Allure] [xxx] Session [xx] total indexed [N], ignored [0], processed [0], errors [N]
These records (indexed is more than zero and processed is zero and errors more than zero) mean that before Allure TestOps agent started there were files in the allure-results directory and no files were added after the agent started. This situation happens for example when you're running your test, collect results copy them to a directory and only then Allure TestOps agents starts.
How to fix
If your test results are copied from somewhere before Allure TestOps agent starts and then no updates happen, you need to index and upload existing files.
For a plug-in (e.g. for Jenkins, TeamCity) you need to add a parameter indexExistingFiles: true to index existing files.
withAllureUpload(indexExistingFiles: true, name: '${SOME_JOB_NAME}', projectId: '', serverId: '', tags: '') {
// some block
}
Missing results - Launch is not empty - Some results are missing - Muted tests
If you or someone from your team marks a test as muted, the results from these tests won't be shown in the Launches.
Muted tests consequences
- Muted test won't appear in the launches main page.
- Nevertheless you will see the test results of a muted test if you jump to the Tree section of a particular launch with a muted test.
- You will see the button Unmute in the test result description panel (on the bottom).
- Muted test won't participate in the analytics and statistics calculations.
- Muted test will be marked as resolved even if no defect is linked to it.
How to check - Dashboards - Mute Trend
Go to your project's dashboard and scroll down to the latest widget called Mute Trend. If on the actual date there is any bars in the view, then you have the muted tests and this could be partially the reason why you see different numbers.
How to check - Test cases - Filter
- Go to the Test cases section of your project.
- Activate filters panel
- Create a filter and set Mute = True
- In the test cases list you will see the list of mutest test cases.
- Each muted test case description will contain information about the person marked this test as muted and the date when that was done.
How to check - Test results - Test description panel
In a test result description panel for a test result in an open launch you will see the active button Unmute.
How to fix
Well, to be honest, you don't need to fix it for if a test is muted, this was done intentionally by someone from your team and mute is one of the ways you resolve failures in your tests.
When the problem with a failing test is fixed, mute can be removed. Otherwise, you can always ask a person that has muted a test, why did they do it.
Missing results - Size of a result file
There is a 2,000,000 bytes limit for a test result file (*-result.json).
If the size of a result file exceeds 2,000,000 bytes, such result file won't be processed but will be considered as an attachment by Allure TestOps. This is done intentionally to avoid failures due to insufficient resources when processing big amount of files.
How to check
Check your allure-results folder to see if there any *result.json files with the size more than 3 megabytes.
How to fix
Consider transferring some of the information you're adding directly into test results to attachments instead. Allure TestOps with the allure framework under the hood allow adding textual information as strings or CSV tables. This will unload the system and for most of the cases will represent the information in a way better than just the information added to test cases' scenario.
Missing results - Errors during the upload
Actually, this one should be the item #1 in the list, but it is what it is.
Tests could have missing results due to the settings of the network equipment (routers, firewalls) and network software (reverse proxy like nginx etc.).
How to check
You either need your network administrator or DevOps or any other person responsible for the network configuration, reverse proxy configuration, firewalls configuration.
You need to check the following
- settings for the reverse proxy (if any) described here
- timeouts
- data transfer limitations
- network timeouts on routers/firewalls similar to reverse proxy timeouts
- network settings on routers/firewalls similar to reverse proxy limits for data transfer size
- blocking rules (black lists, white lists etc.)
How to fix
Try to fix the limits for timeouts and data transfer size to recommended values
Missing test results - Retries
Symptoms: There are several / many *-results.json files for several tests but in the test results of a launch there is just one test result instead.
That could happen if...
- your tests are from different projects (code wise) but have same full qualified name for some reason.
- your test is parameterized but you don't provide parameters in the test results.
How to check
- Go to the test results of a launch Launches => Specific launch => Tree tab
- Select your test case
- In test case's panel go to retries tab
- If your test result has retries and that was only one test run, it seems you need to update your code.
How to fix
If test result has retries and this is not a parameterized test, you need to ensure that full path of your tests differs. There are several different ways to do that:
- Explicitly Assign Allure ID to both of your tests in your code.
- Make sure the test methods are named differently.
- It's better to apply both from above.
If test result has retries and this is a parameterized test, then you need clearly provide the parameters to the test results as some of the test frameworks do not do that by default.
Here is an example for Java:
<snip>
private static final String OWNER = "allure-framework";
private static final String REPO = "allure2";
<snip>
@Story("Create new issue")
@ParameterizedTest(name = "Create issue via api")
@ValueSource(strings = {"First Note", "Second Note"})
public void shouldCreateUserNote(String title) {
parameter("owner", OWNER);
parameter("repo", REPO);
parameter("title", title);
steps.createIssueWithTitle(OWNER, REPO, title);
steps.shouldSeeIssueWithTitle(OWNER, REPO, title);
}
For the example above, the information about the parameters owner, repo and title will land in the test result file in the parameters section:
"parameters": [
{
"name": "owner",
"value": "allure-framework"
},
{
"name": "repo",
"value": "allure2"
},
{
"name": "title",
"value": "First Note"
}
],
These parameters will be processed by Allure TestOps and for the same test case you will see several test results:
... and the then the following test case will be created:
