Allure TestOps - Describing defects

Defects

Main concept. What’s a defect?

Say, you are running a lot of automated tests.

At some point in time you could have many failed tests that are failing with the same cause: there could be certain bug or whatever event that causes tests to fail. And you are pretty certain what causes these failed tests as you have already analysed these, and some actions are already taken (we hope).

In the same time there are other failed tests which have different root cause and you need quickly distinguish between the tests failed with a known reason - defect from the other failed tests which root cause(s) you still need to understand and investigate by analysing the test results.

So, to allow you to easily detect if a test has failed with a known cause, you can link tests to a defect.

Linking failed tests to a defect

To do all the manipulations described in this section, you need to have a Launch in open state. If a launch is already closed, you need to reopen it by clicking Play button.

Opened and closed launches

You can find the results of completed test runs in Launches area of a project.

Where to find the test runs

  1. Jump to a test launch with tests that are failed. In its Tree section click the Filter icon.

  2. Check filter Without resolution

  3. Inevitably you will get the list of tests which have no resolution and contain a description of the things went wrong

    Tests without resolution

  4. Now, let’s say that the error message we get is well known and caused by the ‘Main page header change’. Let’s link this test to this known defect which causes the failed test - click Link defect.

  5. Name the defect (remember, it’s ‘Main page header change’?), then click +Create blah-blah-blah…. Tests without resolution

  6. You will be transferred to more detailed defect description form.

  7. Now, let’s create automation rule that will allow us referencing similar failed tests to this very defect caused by the same bug/issue/event.

    • Click Create automation rule.
  8. Then give the name for this automation rule in Rule name field - (1) in the picture bellow.

  9. Then enter the pattern for the error message. You can use a regular expression in this field - (2) in the picture below.

    Automation rule for a defect

  10. Then to check that your message pattern is actually working, press View matched results button.

    Tests without resolution

    Then go back to the form and finalise the linking of the defect by finally clicking Link the defect button.

Defect is now visible in the launch

  1. Go to the Overview section of a launch and check Defects area of the dashboard. Defect visible in dashboard of the launch
  2. Go to the Launches area of the project and check the launch where you introduced new link to the defect. New defect introduced

Now, the magic

When new failed tests arrive (we hope that won’t happen of course but still), Allure TestOps will automatically link failed tests to a defect if their error messages match the criteria you created to link your test to a defect.

The information about the failed or broken tests that are linked to some defect could be obtained from:

  1. Defects section of a project.
  2. Launches section of a project.
  3. Overview section of a launch.

All these options will be described below.

Defects section of a project

  1. Jump to a project
  2. Jump to defects section
  3. Select registered defect to check in which Launches failed tests with this very defect appear.
  4. In the right side of the screen you will see the list of Launches (test runs) that have tests which failed due to certain defect.

Defects section of a project

Launches section of a project

For all the launches that contain failed tests caused by a defect Allure will now report information that the launch has defects and will show the number of tests that are linked to existing defects.

  1. Jump to a project
  2. Jump to the launches section
  3. For a launch which has newly introduced defect you will see Defects: +1
  4. For the launches with known defects you will see the number of known defects present in the launch

Defects in launches section

Launch overview

  1. Jump to a project
  2. Jump to the launches section
  3. Go to a certain launch you want to analyse.
  4. Refer to defects section for known problems in the launch
  5. Refer to Unresolved test results section for new issues.

Defects in a launch overview

Recap

So, having such a wonderful possibility to assign the tests results to defects in automated way will separate the problems in your tests into two big groups:

  1. Known issues - defects, which are already processed, analysed, checked and considered.
  2. New issues which still need to be analysed.

This will give your team the clear marks to perform certain actions for new items and control the old ones, which ultimately will save you a considerable amount of time.