Running an automated test case

Unlike a manual test which contains human-readable instructions for a tester, an automated test is written as a code. The code not only performs all required actions and validations automatically, but also generates a test result: one or more files describing how long the execution took, what errors were discovered, and so on. Based on this data, Allure TestOps creates or updates the list of test case.

Test results can be uploaded to Allure TestOps via:

• web interface,
• CI integration,
• allurectl utility.

The uploaded data will be reflected in Allure TestOps: first in the current test launch, and eventually (once the launch is closed) in the Test cases.

In this tutorial, we will mostly focus on uploading test results via the web interface. However, you can choose other methods if they better suit your project's workflow.

Where do I get test results from?

The exact path to the test results depends on both the test framework in use and its configuration. When using one of the official Allure adapters and the default settings, test results will typically be generated to the allure-results or build/allure-results directory. The files in the directory are written in the Allure Report data format which is compatible with all the features of Allure TestOps. However, Allure TestOps recognizes other popular formats, too.

If you don't have a working project set up right now, use one from allurereport.org/start to generate a test results directory so that you can follow the tutorial.

Uploading test results manually

In Allure TestOps, a collection of test results is called a launch, and test results are uploaded in the Launches section.

1. Open your project.

2. Go to Launches.

3. In the top right corner, click the Upload new results icon.

   

4. On the left pane of the dialog that appears, fill in the fields:

   • Name — the name that will be displayed in the list of launches.
   • Tags — arbitrary tags for the launch.
   • Issues — links to related issues (if you have configured an issue tracker integration).
   • Links — links to any web pages useful for the team members who will work with the test launch.
   • Environment — values to add to each result, e.g., the name of the browser which was used for testing.

   Except for the Name field, all the fields are optional.

   

   Feel free to establish your own conventions that will help you better navigate through the list of your launches. For example, you may want to include the word “release” in the name of each test launch that you perform before a public release, or set different tags for launches done by different team members. You will be able to search launches by these fields in the web interface or via AQL.

5. On the right pane of the dialog, choose the files to upload — for example, grab an allure-results directory produced by your project's tests and drag it to the upload area. Alternatively, you can compress it into a ZIP archive and then choose it as a single file.

6. Click Upload Files and wait a few moments while the data is processed.

   At this point, the uploaded data are isolated within the test launch. For example, if the data includes a test case that was never before added to Allure TestOps, it won't appear in the Test cases interface just yet.

   If you want to add some more test results to the same test launch (i.e., test results from another machine), go to the launch and select ⋯ → Add test cases → From test result files in the upper-right corner.

   

7. To close the launch, thus updating the Test cases list, go to the launch and click the Stop button.

   

Uploading test results from a CI server

In an actively developed project, tests are often run on a CI server — using GitHub Workflows, GitLab CI, Jenkins, etc. However, manually starting new CI jobs with the required environments and then uploading the results to Allure TestOps can become time-consuming. CI integrations enable a smoother integration of Allure TestOps into your workflow, making it a convenient interface for both starting automated test runs and analyzing their results.

Setting up an integration varies slightly between CI servers, but generally involves two main steps:

1. Configure the CI server to connect with Allure TestOps and upload test results after each test run.

   This may involve installing, authorizing, and configuring a specific plugin or adjusting the test execution command.

2. Configure Allure TestOps to connect to the CI server and trigger a new test run when desired.

   This setup requires some configuration by both the Allure TestOps administrator and the project owner. Once configured, Allure TestOps can send test plans and environment variables to the CI server.

For specific instructions on different CI servers, see the Integrations section.

Once the integration is configured, follow these steps to use it:

1. In the Test cases section, select one or more automated test cases using Ctrl or by checking the boxes next to them.

2. At the bottom of the screen, click Run.

   

3. In the dialog that appears, go to the Job tab and make sure it displays the CI server where you want to run the tests. If necessary, use the Replace job button.

   

4. Click Submit.

   The launch will be created.

5. In the Launches section, find the newly created launch. In the Job runs row below the progress bar, a blinking icon indicates a running CI job. For launches with multiple sets of environment variables, multiple jobs will be started on the CI server, so you’ll see several icons.

   

   After initialization (which may vary in duration depending on the integration), the CI job icon becomes clickable, allowing you to quickly access the job in the CI server interface.

6. Once test execution is complete, you can browse the test results the same way you do in any other test launch.

   Below each test result, there will be a link to the CI job in which the test was executed.

   

What's next?

• Create a combined launch by running manual and automated tests together.
• View the test results in Launch Overview and Dashboards.