Allure TestOps quick start for automation test engineers

Allure TestOps quick start for automation QA

Assumptions

We assume that your aim here is to understand the basic processes, all the advanced stuff like integrations and advanced settings are described in dedicated sections of this site.

The plan

Generally, work with automated tests in Allure TestOps goes as follows:

  1. Create a project.
  2. Add team members.
  3. Work with automated tests
    1. Import test results
    2. Analysis of the test results
    3. Generation of test cases
    4. Run/rerun tests from Allure TestOps

Create project

In Allure TestOps everything goes in a project. No project basically means no work can be done.

In main Allure TestOps UI click + New Project button in upper right corner.

  1. Name your project.
  2. Add comments or description.
  3. Decide whether it should be public or not (see Project visibility below for clarifications).
  4. Hit Submit to start working on your new project.

ALT TEXT

Project visibility

Projects are private by default if not explicitly specified as Public.

Public projects are visible and accessible by every person that has an account in Allure TestOps.

Private projects are not visible for registered users that aren’t added to these as project team members.

Abbreviation

Abbreviation is required now, but it will phase out in upcoming releases.

Adding team members

The person created a project will automatically become an owner of the project.

To add team members to your project do the following:

  1. In a project jump to the Settings.
  2. Select Access.
  3. Either in Groups or in Collaborators section find needed group or a person
  4. Set permission level for new team member (group) and click green plus button.

ALT TEXT

Working with automated tests

Now, when the project and team are set we can start working with automated tests.

As said before, first we need to import the test results. We’ll describe the automation of manual tests and generally creation of the automated tests in connection with Allure TestOps in [special section].

Say, you have written bunch of test in your favourite test framework and they are running on your CI and so what? You need to provide the test results to a person that needs these results in consumable format. You need to keep in mind this person won’t worry about looking into details as they are probably not tech savvy.

What shall we do?

Import test results to Allure TestOps

Why?

Allure will generate the test documentation and keep it updated based on the test results you provide to it.

Allure will ease the problems resolution process, i.e. will save your time (time = money, right?).

Allure will allow you analyse the test results by creating dashboards and widgets with the analytics, thus will show you the problem areas, ineffective tests and problematic ones, i.e. same you will save time and money.

How?

Assuming someone in your company already uses Allure TestOps, you just need to show to Allure TestOps where the tests results reside, tell what’s the name of the project and plug-in or command line tool will send the test results to your project.

CI plug-in configuration

Depending on the CI you’re using you’ll have settings that you need to make in different (alas!) places but generally you’ll have to specify the following:

  1. ID of your Allure TestOps server as per the configuration of your CI server.
  2. Project name to which you need to send the test results.
  3. Tags for launches.
  4. Template for launches names.
  5. Real path, where you store the test results in scope of your build job.
Example for Jenkins freestyle project

ALT TEXT

Example for Jenkins pipeline

Check the line that contains withAllureUpload.

pipeline {
    agent any
    stages {
        stage('Checkout') {
            steps {
                git 'https://github.com/eroshenkoam/allure-example.git'
            }
        }
        stage('Build') {
            steps {
                withAllureUpload(serverId: 'main', projectId: '50', tags: 'filter', results: [[path: 'build/allure-results']]) {
                    withAllureTestPlan(filter: 'status = "Active"') {
                        sh './gradlew clean test'
                    }                    
                }                
            }
        }
    }
}
Example for TeamCity

Allure TestOps’ plug-in settings can be found in Settings » Build features of your TeamCity’s build job.

ALT TEXT

Example for gitlab
stages:
  - test

variables:
  ALLURE_LAUNCH_NAME: "${CI_PROJECT_NAME} - ${CI_COMMIT_SHORT_SHA}"
  ALLURE_LAUNCH_TAGS: "regular, ${CI_COMMIT_REF_NAME}, ${GITLAB_USER_EMAIL}"
  ALLURE_TESTPLAN_PATH: ./testplan.json

test:
  image: gradle
  stage: test
  before_script:
    - wget https://github.com/allure-framework/allurectl/releases/download/1.18.1/allurectl_linux_386 -O /usr/bin/allurectl
    - chmod +x /usr/bin/allurectl
    - allurectl job-run start
    - allurectl job-run plan --output-file ${ALLURE_TESTPLAN_PATH}
  script:
    - allurectl upload --job-run-child --timeout 6000 allure-results &
    - echo "Running tests on ${BASE_URL} - ${BROWSER_NAME}"
    - gradle test
    - pkill allurectl
  after_script:
    - allurectl job-run stop

Run your build job

From whatever CI system, after the configuration of Allure TestOps agent (plug-in or allurectl) is performed, just start your build job and jump to your project in Allure TestOps to the section Launches.

You will see your launch’s progress:

ALT TEXT

Then you will see that the test results starting to arrive from your CI to Allure TestOps.

If the executor’s icon stopped blinking, then all of your test results arrived. You can jump to just finished launch:

Analysis of test results

Analysis of the tests results goes in the tabs of a Launch. To start the review of a Launch, select it in the list of launches from the main UI of a project.

Overview tab

Overview will show you basic analysis for a launch.

ALT TEXT

  1. Success rate widget shows the balance between passed and failed tests.
  2. Job runs shows the number of the job runs within current Launch, there could be several job runs if you made retries for some of the tests (will discuss that later on).
  3. Features and Suites widgets show the distribution of tests between the trees for Features or Suites defined by default for each project and their status (passed, broken, failed). This could be extended in project settings.
  4. Defects will show you the failed tests that your team has already analysed and detected the root cause of failure and maybe even the faulty part - code or the test, and defined a way to fix the failure. Defects are defined manually and assigned by Allure TestOps in automated way based on the created rules.
  5. Unresolved test results widget contains all the test results that are either failed or broken and aren’t assigned to any defect, i.e. here you will see the test results you still need to analyse.
  6. Retries widget will contain the test cases you’ve retried in scope of current Launch.

Tree tab

Tree tab shows the test results in a tree structure defined for each project. By default there are two trees defined by the system - Suites (1) and Features (2) and a flat view (3).

ALT TEXT

In the tree tab you can check all the test results received from your CI system by selecting a test case in a tree (4).

If no additional settings has been made then each test will contain the basic information at least about the Suite a test belongs to, steps (5), status (6), execution history (7), owner of the test if defined in the test cases code. Tree tab, or more specifically the test case area allow you to link a failed or broken test to a defect, assign mute to failed or broken tests and rerun the test to check if the failure was just random event or there is a real problem with it.

Errors tab

Errors tab will aggregate all the errors appeared in current Launch and will allow you to analyse the error messages, trace and assign a defect (1) to a certain error, mute (3) a test or rerun (3) test with errors.

As said before a defect means that you’ve analysed and understood the problem and further analysis of similar errors is not needed.

Mute will allow ignoring tests with similar errors if they are assigned to a certain issue in an issue tracker. Tests will arrive but the results won’t be shown unless test is unmuted.

Rerun allows you to run a test again to check if the failure was not random but caused by some real error.

ALT TEXT

Graphs and Timeline tabs

These tabs will show you how your tests have been running over the time.

Generation of test cases

Test cases or test documentation or Live documentation is generated based on the test results. To generate the documentation for your tests you need to close a Launch.

Closing a Launch

Two ways:

  • manual closing: just push the button. ALT TEXT
  • automated closing, settings either on global or on project’s level ALT TEXT

Test cases will be generated right on the closure of the launch. Alongside with the generation of the test cases Allure TestOps will count the test result for closed launch in the statistics and they will be shown in all the analytics dashboards/widgets.

Statistics / analytics / are considered as advanced study and described in [dedicated sections].

Run / rerun jobs and test cases from Allure TestOps

There are several scenarios here to run tests from Allure TestOps side:

  1. Run full CI job.
  2. Run a set of tests.
  3. Run a test plan
  4. Rerun tests failed, broken or whatever tests from an open Launch.

Run full CI job

Prerequisites

To run entire build job existing on CI server from Allure TestOps you need to run CI job it at least once on CI side and receive the test results.

Running a job
  1. To run entire job in your project jump to Jobs section of your project select an existing job and press run button (first button for a Job).

ALT TEXT

  1. Do not change anything in next modal window but just press Submit.
  2. Then you can jump to the Launches section to check the status of your launch.

Run a set of tests

  1. Jump to the test cases section select the test cases in a tree.
  2. Select test cases you want to run on CI side.
  3. Click on bulk actions button.
  4. Select Run

ALT TEXT

  1. Do not change anything in next modal window but just press Submit.
  2. Then you can jump to the Launches section to check the status of your launch.

Run a test plan

If you haven’t any test plans, then let’s create one.

Creating a test plan

There are several ways, but let’s settle on creating from Test cases section.

  1. In test cases section enable filtering.
  2. Apply some filter (e.g. by the tag ‘smoke’).
  3. Select all test cases appearing after you’ve applied the filter.
  4. Click on bulk actions
  5. Create your test plan.

ALT TEXT

Running a test plan
  1. Jump to Test plans section.
  2. Click Run for the test plan of your choice.
  3. Don’s change anything and click submit.
  4. Then you can jump to the Launches section to check the status of your launch.

Rerun tests

Why rerun?

Say, you’re not sure if a test is really faulty or broken and you need to test your hypothesis. Or you could have some other reason.

Rerun
  1. Jump to launches.
  2. In an open launch enable filter for tests.
  3. Apply the filter to select Failed test only.
  4. Select all failed tests, apply bulk action Rerun.
  5. Use the parameters suggested by Allure.
  6. Then you can jump to the Launches section to check the status of your launch.

Summary

These are the basic actions you will need when starting your way with Allure TestOps.

For advanced actions please use our other guides and FAQ.