Selective test run (or test cases filtering)
What is selective test run (or test filtering) and why it is brilliant?
Prerequisites
To allow test cases selective run an allure framework adaptor or your own adaptor for your test framework should exist. If you are not sure if your test framework has support for selective test run or you are 100% sure it is not implemented, please create an Feature request at our help desk for the development for this feature.
Checking locally if the selective run is working for you
Important: this will work if the integration with your tests is done with officially supported Allure Framework integration from our repositories., if you are using a 3rd party adaptor, then you need to communicate with the original developer.
Briefly
This section describes really briefly, how the selective test running is working.
- Based on your inputs (selected test cases or test plan) Allure Testops creates a list of test cases containing the IDs of all selected test cases and their selectors (selector is the full name of the test generally containing the combination of package, class and method name of a test) and sends API command to start a build job to your CI system.
- Allure Testops' agent (that could be either plug-in or allurectl tool) is being triggered by a start of a build job and saves this list as testplan.json file, and creates/ an environment variable ALLURE_TESTPLAN_PATH and the path to testplan.json is assigned to this newly created variable.
- Then Allure Testops' agent initiates the execution of the tests.
- Existing integration with a test framework reads the information from testplan.json and provides this information to test framework so it's able to execute the tests from testplan.json only ignoring the other tests in your project.
Example. Failed or broken tests rerun
To perform this operation, the launch's state must be set to "Open"
Say you have around 100 of tests in your test run. Imagine five of these tests either failed during the the launch or appeared to be broken. Normally, to recheck such tests you either need to rerun full bunch of 100 test cases or make the manual test run for the failed tests. Both options are either long or boring or even both.
Now, we need to reopen the launch and then jump inside the launch to the test cases tree
Now, if you jump to the Launches section of your project and find the launch for with you just selected and started rerun job, you will see that there is completed run and an active one as well as part of the test cases are completed and only one is running (see the picture below).
If you look into the logs of CI system's job run (in our case this is Jenkins' example), you will see that Allure TesOps' agent received tesplan.json with one test inside.
If the tests you are rechecking/rerunning were indeed randomly failed or broken due to any reason, you will see that the launch will now contain all tests as passed with two successfully completed job runs.