Allure Framework is a flexible lightweight multi-language test report tool that not only shows a very concise representation of what have been tested in a neat web report form, but allows everyone participating in the development process to extract maximum of useful information from everyday execution of tests. From the dev/qa perspective Allure reports shorten common defect lifecycle: test failures can be divided on bugs and broken tests, also logs, steps, fixtures, attachments, timings, history and integrations with TMS and bug-tracking systems can be configured, so the responsible developers and testers will have all information at hand. From the managers perspective Allure provides a clear 'big picture' of what features have been covered, where defects are clustered, how the timeline of execution looks like and many other convenient things. Modularity and extensibility of Allure guarantees that you will always be able to fine-tune something to make Allure suit you better.

1. About

The Allure reference guide is available as html documents. The latest copy is available at https://docs.qameta.io/allure/latest

Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.

1.2. Get help

There are several places to getting help:

  • Contact the community on Gitter. Also we have separate russian room

  • Ask a question on stackoverflow.com, the tag is allure

  • Report bugs in github issues

1.3. How to proceed

  • Open the demo version, if you just want to see what the report looks like

  • Go to Get started to build report for existing project

  • Learn more about report structure and features in this section

  • Choose your favorite testing framework and integrate it with Allure. Supported frameworks are divided into language-based sections: Java, Python, JavaScript, Ruby, Groovy, PHP, .Net, Scala.

2. Get Started

To generate your first report you will need to go through just a few simple steps:

  • Downloading and installing Allure commandline application suitable for your environment.

  • Locating test execution data that you have to build a report on.

2.1. Installing a commandline

Several options for Allure installation are currently supported:

2.1.1. Linux

For debian-based repositories a PPA is provided:

sudo apt-add-repository ppa:qametaio/allure-framework #not published yet
sudo apt-get update 
sudo apt-get install allure-commandline

2.1.2. Mac OS X

For Mas OS, automated installation is available via Homebrew

brew install allure

2.1.3. Windows

For Windows, Allure is available from the Scoop commandline-installer.

To install Allure, download and install Scoop and then execute in the Powershell:

scoop install allure

Also Scoop is capable of updating Allure distribution installations. To do so navigate to the Scoop installation directory and execute

\bin\checkver.ps1 allure -u

This will check for newer versions of Allure, and update the manifest file. Then execute

scoop update allure

to install a newer version. (documentation)

2.1.4. Manual installation

  1. Download the latest version as zip archive from bintray.

  2. Unpack the archive to allure-commandline directory.

  3. Navigate to bin directory.

  4. Use allure.bat for Windows or allure for other Unix platforms.

  5. Add allure to system PATH.

To run commandline application, Java Runtime Environment must be installed.

2.1.5. Check the installation

Execute allure --version in console to make sure that allure is now available:

$ allure --version
2.0.1

2.2. Test execution

Before building a report you need to run your tests to obtain some basic test report data. Typically it might be a junit-style xml report generated by nearly every popular test framework. For example, suppose you have test reports automatically created by surefire maven plugin stored in the target/surefire-reports:

Surefire report folder

2.3. Report generation

This is already enough to see the Allure report in one command:

allure serve /home/path/to/project/target/surefire-reports/

Which generates a report in temporary folder from the data found in the provided path and then creates a local Jetty server instance, serves generated report and opens it in the default browser. It is possible to use a --profile option to enable some pre-configured allure setting. junit profile is enabled by default, you will learn more about profiles in the following section.

This would produce a report with a minimum of information extracted from the xml data that will lack nearly all of the advanced allure features but will allow you to get a nice visual representation of executed tests.

Report generated on xml data

3. Report structure

Once you’ve got the idea what the report does look like. You will probably want to get more data-rich reports. You might have to consider using one of the Allure adaptors for your testing framework, which will allow to collect much more information. Jump to the integrations section to learn more about integration with testing frameworks.

Typical report consists of 'Overview' tab, navigation bar, several tabs for different kinds of test data representation and test case pages for each individual test. Each Allure report is backed by a tree-like data structure, that represents a test execution process. Different tabs allow to switch between the views of the original data structure thus giving a different perspective. Note that all tree-like representations including Behaviors, Categories, xUnit and Packages support filtering and are sortable.

3.1. Overview page

Entry point for every report would be the 'Overview' page with dashboards and widgets:

Overview

Overview page hosts several default widgets representing basic characteristics of your project and test environment.

  • Statistics - overall report statistics

  • Launches - if this report represents several test launches, statistics per launch will be shown here

  • Behaviors - information on results aggregated according to stories and features

  • Executors - information on test executors that were used to run the tests

  • History Trend - if tests accumulated some historical data, it’s trend will be calculated and shown on the graph.

Home page widgets are draggable and configurable. Also, Allure supports it’s own plugin system, so quite different widget layouts are possible.

Navigation bar is collapsible and enables you to switch into several of the basic results overview modes.

3.2. Categories

Categories tab gives you the way to create custom defects classification to apply for test results.

Categories

3.3. Suites

On the Suites tab a standard structural representation of executed tests, grouped by suites and classes can be found.

Suites

3.4. Graphs

Graphs allow you to see different statistics collected from the test data: statuses breakdown or severity and duration diagrams.

Graphs

3.5. Timeline

Timeline tab visualizes retrospective of tests execution, allure adaptors collect precise timings of tests, and here on this tab they are arranged accordingly to their sequential or parallel timing structure.

Timeline

3.6. Behaviors

For Behavior-driven approach, this tab groups test results according to Epic, Feature and Story tags.

Behaviors

3.7. Packages

Packages tab represents a tree-like layout of test results, grouped by different packages.

Packages

3.8. Test case page

From some of the results overview pages described above you can go to the test case page after clicking on the individual tests. This page will typically contain a lot of individual data related to the test case: steps executed during the test, timings, attachments, test categorization labels, descriptions and links.

Test result page

4. Features

This section describes the main features of Allure. For example, you can group your tests by stories or features, attach files, and distribute assertions over a set of custom steps, among other features. All features are supported by Java test frameworks, so we only provide Java examples here. For details on how a particular adapter works with the test framework of your choice, refer to the adapter guide.

4.1. Flaky tests

In real life not all of your tests are stable and always green or always red. A test might start to "blink" i.e. it fails from time-to-time without any obvious reason. You could disable such a test, that is a trivial solution. However what if you do not want to do that? Say you would like to get more details on possible reasons or the test is so critical that even being flaky it provides helpful information? You have an option now to mark such tests in a special way, so the resulting report will clearly show them as unstable:

@Flaky
public void aTestWhichFailsFromTimeToTime {
     ...
}

Here is what you get in the report if such a test failed:

A failed test marked as flaky
you can mark a whole test class as flaky as well.

5. Java

5.1. jUnit 4

5.1.1. Installation

The latest available version of allure-junit4:

Allure JUnit4
Maven

Add the following to your pom.xml:

pom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
<properties>
    <aspectj.version>1.8.10</aspectj.version>
</properties>

<dependencies>
    <dependency>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-junit4</artifactId>
        <version>LATEST_VERSION</version>
        <scope>test</scope>
    </dependency>
</dependencies>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.20</version>
            <configuration>
                <testFailureIgnore>false</testFailureIgnore>
                <argLine>
                    -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
                </argLine>
                <properties>
                    <property>
                        <name>listener</name>
                        <value>io.qameta.allure.junit4.AllureJunit4</value>
                    </property>
                </properties>
            </configuration>
            <dependencies>
                <dependency>
                    <groupId>org.aspectj</groupId>
                    <artifactId>aspectjweaver</artifactId>
                    <version>${aspectj.version}</version>
                </dependency>
            </dependencies>
        </plugin>
    </plugins>
</build>
...

Then run the build as usual:

$ mvn clean test
Gradle

For Gradle users, the allure-gradle plugin is available. The plugin autoconfigures all of the required dependencies:

build.gradle
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
buildscript {
    repositories {
        jcenter()
    }
    dependencies {
        classpath "io.qameta.allure:allure-gradle:2.3"
    }
}

plugins {
    id 'io.qameta.allure'
}

allure {
    version = '2.2.1'
    autoconfigure = true
    aspectjweaver = true
    allureJavaVersion = LATEST_VERSION
}

Then run the build as usual:

$ ./gradlew clean test

Allure results will appear in build/allure-results folder. To generate html report and automatically open it in a web browser, run the following command:

$ ./gradlew allureServe build/allure-results

5.1.2. Features

Java annotations and traits are available to use the main Allure features.

DisplayName

In order to add a human-readable name to any test method, annotate it with @DisplayName annotation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
package my.company.tests;

import org.junit.Test;
import io.qameta.allure.junit4.DisplayName;

public class MyTests {

    @Test
    @DisplayName("Human-readable test name")
    public void testSomething() throws Exception {
        //...
    }

}
Description

Similarly you can add detailed description for each test method. To add such description use @Description annotation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
package my.company.tests;

import org.junit.Test;
import io.qameta.allure.Description;

@Test
public class MyTests {

    @Test
    @Description("Some detailed test description")
    public void testSomething() throws Exception {
        ...
    }

}
Steps

Steps are any actions that constitute a testing scenario. Steps can be used in different testing scenarios. They can: be parametrized, make checks, have nested steps, and create attachments. Each step has a name.

In order to define steps in Java code, you need to annotate the respective methods with @Step annotation. When not specified, the step name is equal to the annotated method name converted to human-readable format.

Note that steps' mechanics was revised and now it supports smart fields' analysis. In Allure 1 users had to specify indexes to refer which args' values they want to inject into step. Allure 2 uses reflection-based approach, which provides deep fields' extraction by their names.

Assuming that you have the following entity:

1
2
3
4
5
6
public class User {

     private String name;
     private String password;
     ...
}

You can access these fields' values directly by name:

1
2
3
4
5
6
7
8
import io.qameta.allure.Step;

...

@Step("Type {user.name} / {user.password}.")
public void loginWith(User user) {
     ...
}

Arrays and Collections are supported as well. So you don’t need to explicitly override toString() for your custom objects anymore.

Attachments

An attachment in Java code is simply a method annotated with @Attachment that returns either a String or byte[], which should be added to the report:

1
2
3
4
5
6
7
8
9
10
11
12
13
import io.qameta.allure.Attachment;

...

@Attachment
public String performedActions(ActionSequence actionSequence) {
    return actionSequence.toString();
}

@Attachment(value = "Page screenshot", type = "image/png")
public byte[] saveScreenshot(byte[] screenShot) {
    return screenShot;
}

Or you can use Allure helper methods

1
2
3
4
5
6
7
8
9
10
import io.qameta.allure.Allure;

...

Allure.addAttachment("My attachment", "My attachment content");

Path content = Paths.get("path-to-my-attachment-contnet");
try (InputStream is = Files.newInputStream(content)) {
    Allure.addAttachemnt("My attachment", is);
}
If return type in a method annotated with @Attachment differs from String or byte[] we call toString() on return value to get attachment contents.

You can specify exact MIME type for each attached file using type parameter of @Attachment annotation like shown above. However there’s no need to explicitly specify attachment type for all attached files as Allure by default analyses attachment contents and can determine attachment type automatically. You usually need to specify attachment type when working with plain text files.

ou can link your tests to some resources such as TMS (test management system) or but tracker.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
import io.qameta.allure.Link;
import io.qameta.allure.Issue;
import io.qameta.allure.TmsLink;

@Link("https://example.org")
@Link(name = "allure", type = "mylink")
public void testSomething() {
     ...
}

@Issue("123")
@Issue("432")
public void testSomething() {
     ...
}

@TmsLink("test-1")
@TmsLink("test-2")
public void testSomething() {
     ...
}

In order to specify the link pattern you can use the system property in the following format: allure.link.my-link-type.pattern=https://example.org/custom/{}/path. Allure will replace {} placeholders with value specified in annotation. For example:

allure.link.mylink.pattern=https://example.org/mylink/{}
allure.link.issue.pattern=https://example.org/issue/{}
allure.link.tms.pattern=https://example.org/tms/{}
Severity

@Severity annotation is used in order to prioritize test methods by severity:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
package org.example.tests;

import org.junit.Test;
import io.qameta.allure.Severity;
import io.qameta.allure.SeverityLevel;

public class MyTest {

    @Test
    @Severity(SeverityLevel.CRITICAL)
    public void testSomething() throws Exception {
        // ...
    }

}
Behaviours Mapping

In some development approaches tests are classified by Features and Stories. To add such mapping you can use Epic, Feature and Stories annotations:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
package org.example.tests;

import org.junit.Test;
import io.qameta.allure.Epic;
import io.qameta.allure.Feature;
import io.qameta.allure.Story;

@Epic("Allure examples")
@Feature("Junit 4 support")
public class MyTest {

    @Test
    @Story("Base support for bdd annotations")
    @Story("Advanced support for bdd annotations")
    public void testSomething() throws Exception {
        // ...
    }

}

5.2. TestNG

5.2.1. Installation

The available latest version of allure-testng:

Allure TestNG
Maven

You need to add the following to your pom.xml:

pom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
<properties>
    <aspectj.version>1.8.10</aspectj.version>
</properties>

<dependencies>
    <dependency>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-testng</artifactId>
        <version>LAST_VERSION</version>
        <scope>test</scope>
    </dependency>
</dependencies>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.20</version>
            <configuration>
                <argLine>
                    -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
                </argLine>
            </configuration>
            <dependencies>
                <dependency>
                    <groupId>org.aspectj</groupId>
                    <artifactId>aspectjweaver</artifactId>
                    <version>${aspectj.version}</version>
                </dependency>
            </dependencies>
        </plugin>
    </plugins>
</build>

Then run the build as usual:

$ mvn clean test

Allure results will appear in target/allure-results folder. To generate html report and automatically open it in a web browser, run the following command:

$ allure serve target/allure-results
Gradle

For Gradle users, the allure-gradle plugin is available. The plugin autoconfigures all of the required dependencies:

build.gradle
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
buildscript {
    repositories {
        jcenter()
    }
    dependencies {
        classpath "io.qameta.allure:allure-gradle:2.3"
    }
}

plugins {
    id 'io.qameta.allure'
}

allure {
    version = '2.2.1'
    autoconfigure = true
    aspectjweaver = true
    allureJavaVersion = LATEST_VERSION
}

Then run the build as usual:

$ ./gradlew clean test

Allure results will appear in build/allure-results folder. To generate html report and automatically open it in a web browser, run the following command:

$ ./gradlew allureServe build/allure-results

5.2.2. Features

This adapter comes with a set of Java annotations and traits allowing to use main Allure features.

DisplayName

In order to add human-readable name to any test method you need to use description property of @Test annotation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
package my.company.tests;

import org.testng.annotations.Test;

public class MyTests {

    @BeforeMethod(description = "Configure something before test")
    public void setUp() {
        //...
    }

    @Test(description = "Human-readable test name")
    public void testSomething() throws Exception {
        //...
    }

}
Description

Similarly you can add detailed description for each test method. To add such description use @Description annotation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
package my.company.tests;

import org.junit.Test;
import io.qameta.allure.Description;

@Test
public class MyTests {

    @Test
    @Description("Some detailed test description")
    public void testSomething() throws Exception {
        ...
    }

}
Steps

Steps are any actions that constitute a testing scenario. Steps can be used in different testing scenarios. They can: be parametrized, make checks, have nested steps, and create attachments. Each step has a name.

In order to define steps in Java code, you need to annotate the respective methods with @Step annotation. When not specified, the step name is equal to the annotated method name converted to human-readable format.

Note that steps' mechanics was revised and now it supports smart fields' analysis. In Allure 1 users had to specify indexes to refer which args' values they want to inject into step. Allure 2 uses reflection-based approach, which provides deep fields' extraction by their names.

Assuming that you have the following entity:

1
2
3
4
5
6
public class User {

     private String name;
     private String password;
     ...
}

You can access these fields' values directly by name:

1
2
3
4
5
6
7
8
import io.qameta.allure.Step;

...

@Step("Type {user.name} / {user.password}.")
public void loginWith(User user) {
     ...
}

Arrays and Collections are supported as well. So you don’t need to explicitly override toString() for your custom objects anymore.

Attachments

An attachment in Java code is simply a method annotated with @Attachment that returns either a String or byte[], which should be added to the report:

1
2
3
4
5
6
7
8
9
10
11
12
13
import io.qameta.allure.Attachment;

...

@Attachment
public String performedActions(ActionSequence actionSequence) {
    return actionSequence.toString();
}

@Attachment(value = "Page screenshot", type = "image/png")
public byte[] saveScreenshot(byte[] screenShot) {
    return screenShot;
}

Or you can use Allure helper methods

1
2
3
4
5
6
7
8
9
10
import io.qameta.allure.Allure;

...

Allure.addAttachment("My attachment", "My attachment content");

Path content = Paths.get("path-to-my-attachment-contnet");
try (InputStream is = Files.newInputStream(content)) {
    Allure.addAttachemnt("My attachment", is);
}
If return type in a method annotated with @Attachment differs from String or byte[] we call toString() on return value to get attachment contents.

You can specify exact MIME type for each attached file using type parameter of @Attachment annotation like shown above. However there’s no need to explicitly specify attachment type for all attached files as Allure by default analyses attachment contents and can determine attachment type automatically. You usually need to specify attachment type when working with plain text files.

ou can link your tests to some resources such as TMS (test management system) or but tracker.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
import io.qameta.allure.Link;
import io.qameta.allure.Issue;
import io.qameta.allure.TmsLink;

@Link("https://example.org")
@Link(name = "allure", type = "mylink")
public void testSomething() {
     ...
}

@Issue("123")
@Issue("432")
public void testSomething() {
     ...
}

@TmsLink("test-1")
@TmsLink("test-2")
public void testSomething() {
     ...
}

In order to specify the link pattern you can use the system property in the following format: allure.link.my-link-type.pattern=https://example.org/custom/{}/path. Allure will replace {} placeholders with value specified in annotation. For example:

allure.link.mylink.pattern=https://example.org/mylink/{}
allure.link.issue.pattern=https://example.org/issue/{}
allure.link.tms.pattern=https://example.org/tms/{}
Severity

@Severity annotation is used in order to prioritize test methods by severity:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
package org.example.tests;

import org.junit.Test;
import io.qameta.allure.Severity;
import io.qameta.allure.SeverityLevel;

public class MyTest {

    @Test
    @Severity(SeverityLevel.CRITICAL)
    public void testSomething() throws Exception {
        // ...
    }

}
Behaviours Mapping

In some development approaches tests are classified by Features and Stories. To add such mapping you can use Epic, Feature and Stories annotations:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
package org.example.tests;

import org.junit.Test;
import io.qameta.allure.Epic;
import io.qameta.allure.Feature;
import io.qameta.allure.Story;

@Epic("Allure examples")
@Feature("Junit 4 support")
public class MyTest {

    @Test
    @Story("Base support for bdd annotations")
    @Story("Advanced support for bdd annotations")
    public void testSomething() throws Exception {
        // ...
    }

}

5.3. Cucumber JVM

5.3.1. Installation

The available latest version of allure-testng:

Allure CucumberJVM
Maven

Simply add allure-cucumber-jvm plugin as dependency to your project and add it to CucumberOptions:

pom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
<properties>
    <aspectj.version>1.8.10</aspectj.version>
</properties>

<dependencies>
    <dependency>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-cucumber-jvm</artifactId>
        <version>LATEST_VERSION</version>
    </dependency>
</dependencies>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.20</version>
            <configuration>
                <argLine>
                    -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
                    -Dcucumber.options="--plugin io.qameta.allure.cucumberjvm.AllureCucumberJvm"
                </argLine>
            </configuration>
            <dependencies>
                <dependency>
                    <groupId>org.aspectj</groupId>
                    <artifactId>aspectjweaver</artifactId>
                    <version>${aspectj.version}</version>
                </dependency>
            </dependencies>
        </plugin>
    </plugins>
</build>

Then execute mvn clean test goal. After tests executed allure JSON files will be
placed in allure-results directory by default. It can be overridden by setting allure.results.directory
system property.

5.3.2. Features

This adapter provides runtime integration allowing conversion of Gherkin dsl features into basic Allure features

Display Name

Titles for tests and suites are extracted at runtime from .feature files

Description

Feature’s description appears on every scenario

Steps

All scenario steps are automatically translated into allure steps

Attachments

An attachment in Java code is simply a method annotated with @Attachment that returns either a String or byte[], which should be added to the report:

1
2
3
4
5
6
7
8
9
10
11
12
13
import io.qameta.allure.Attachment;

...

@Attachment
public String performedActions(ActionSequence actionSequence) {
    return actionSequence.toString();
}

@Attachment(value = "Page screenshot", type = "image/png")
public byte[] saveScreenshot(byte[] screenShot) {
    return screenShot;
}

Or you can use Allure helper methods

1
2
3
4
5
6
7
8
9
10
import io.qameta.allure.Allure;

...

Allure.addAttachment("My attachment", "My attachment content");

Path content = Paths.get("path-to-my-attachment-contnet");
try (InputStream is = Files.newInputStream(content)) {
    Allure.addAttachemnt("My attachment", is);
}
If return type in a method annotated with @Attachment differs from String or byte[] we call toString() on return value to get attachment contents.

You can specify exact MIME type for each attached file using type parameter of @Attachment annotation like shown above. However there’s no need to explicitly specify attachment type for all attached files as Allure by default analyses attachment contents and can determine attachment type automatically. You usually need to specify attachment type when working with plain text files.

To pass issues to report, just add @issue=<ISSUE-NUMBER> on top of Scenario on Feature in your .feature file.

To pass TMS links to report, just add @tmsLink=<TEST-CASE-ID> on top of Scenario on Feature in your .feature file.

do not forget to configure allure properties with link patterns.
Severity

To set severity, add @severity=blocker on top of Scenario on Feature in your .feature file.
If severity has wrong value it will be forced to normal (default).

Supported severity values: blocker, critical, normal, minor, trivial.

Test markers

Every Feature or Scenario can be annotated by following tags: @flaky, @muted, @known

Test fixtures

All methods annotated by @import cucumber.api.java.After or @cucumber.api.java.Before annotations
will appear in the report as steps with method names.
If @Before execution fails, the scenario will be marked as skipped.
If @After execution fails, the scenario will be marked as passed, and only After method’s step
will be marked as failed.

Behaviours Mapping

Allure Cucumber JVM integration uses information extracted from Feature: section.

6. Python

6.1. Pytest

This plugin currently supports only Allure 1.4.x series.

6.1.1. Installation and Usage

py.test --alluredir [path_to_report_dir]
# WARNING [path_to_report_dir] will be purged at first run

This plugin gets automatically connected to py.test via entry point if installed.

Connecting to IDE:

pytest_plugins = 'allure.pytest_plugin',\

6.1.2. Features

Title
Not implemented now
Description
Not implemented now
Steps

To divide a test into steps:

import pytest

def test_foo():
    with pytest.allure.step('step one'):
        # do stuff

    with pytest.allure.step('step two'):
        # do more stuff

Can also be used as decorators. By default step name is generated from method name:

import pytest

@pytest.allure.step
def make_test_data_foo():
    # do stuff

def test_foo():
    assert make_some_data_foo() is not None

@pytest.allure.step('make_some_data_foo')
def make_some_data_bar():
    # do another stuff

def test_bar():
    assert make_some_data_bar() is not None

Steps can also be used without pytest. In that case instead of pytest.allure.step simply use allure.step:

import allure

@allure.step('some operation')
def do_operation():
    # do stuff

allure.step decorator supports step name formatting with function parameters:

import allure

@allure.step('some operation for bar={0}')
def do_operation(bar):
    # do stuff

def test_foo():
    assert do_operation('abcdef')

The step in the latter case will have name some operation for bar=abcdef. Formatting is done via python’s built-in string.format and supports it’s options. Arguments are passed to format method in the same way they are passed to the decorated function.

Steps support is limited when used with fixtures.

Attachments

To attach some content to test report:

import allure

def test_foo():
    allure.attach('my attach', 'Hello, World')
Issues Tracker

Issues can be set for test.

import pytest

@pytest.allure.issue('http://jira.lan/browse/ISSUE-1')
def test_foo():
    assert False
import allure

@allure.issue('http://jira.lan/browse/ISSUE-2')
class TestBar:

    # test will have ISSUE-2, ISSUE-3 and ISSUE-4 label
    @allure.issue('http://jira.lan/browse/ISSUE-3')
    def test_bar1(self):
        # You can use this feature like a function inside the test
        allure.dynamic_issue('http://jira.lan/browse/ISSUE-4')
        pass

    # test will have only ISSUE-2 label
    def test_bar2(self):
        pass
Test Management System

Test cases links can be set for test also.

import pytest

@pytest.allure.testcase('http://my.tms.org/TESTCASE-1')
def test_foo():
    assert False
import allure

@allure.testcase('http://my.tms.org/browse/TESTCASE-2')
class TestBar:

    # test will have TESTCASE-2 and TESTCASE-3 label
    @allure.testcase('TESTCASE-3')
    def test_bar1(self):
        pass

    # test will have only TESTCASE-2 label
    def test_bar2(self):
        pass
Parameters
Not implemented now
Severity

Any test, class or module can be marked with different severity:

import pytest

@pytest.allure.severity(pytest.allure.severity_level.MINOR)
def test_minor():
    assert False


@pytest.allure.severity(pytest.allure.severity_level.CRITICAL)
class TestBar:

    # will have CRITICAL priority
    def test_bar(self):
        pass

    # will have BLOCKER priority via a short-cut decorator
    @pytest.allure.BLOCKER
    def test_bar(self):
        pass

To run tests with concrete priority:

py.test my_tests/ --allure_severities=critical,blocker
Behaviours Mapping

Feature and Story can be set for test.

import allure


@allure.feature('Feature1')
@allure.story('Story1')
def test_minor():
    assert False


@allure.feature('Feature2')
@allure.story('Story2', 'Story3')
@allure.story('Story4')
 class TestBar:

    # will have 'Feature2 and Story2 and Story3 and Story4'
    def test_bar(self):
        pass

To run tests by Feature or Story:

py.test my_tests/ --allure_features=feature1,feature2
py.test my_tests/ --allure_features=feature1,feature2 --allure_stories=story1,story2
Environment Parameters

You can provide test environment parameters such as report name, browser or test server address to allure test report.

import allure
import pytest


def pytest_configure(config):
    allure.environment(report='Allure report', browser=u'Я.Браузер')


@pytest.fixture(scope="session")
def app_host_name():
    host_name = "my.host.local"
    allure.environment(hostname=host_name)
    return host_name


@pytest.mark.parametrize('country', ('USA', 'Germany', u'Россия', u'Япония'))
def test_minor(country):
    allure.environment(country=country)
    assert country

More details about allure environment you can know from https://github.com/allure-framework/allure-core/wiki/ Environment[documentation].

6.1.3. Development

Use allure.common.AllureImpl class to bind your logic to this adapter.

6.2. Behave

Allure integrates with behave as an external formatter.

6.2.1. Instalation

$ pip install allure-behave

6.2.2. Usage

You can specify the formatter directly in the command line:

$ behave -f allure_behave.formatter:AllureFormatter -o %allure_result_folder% ./features

6.2.3. Features

Severity

Tags that are matched to severity names (like critical, trivial, etc.) will be interpreted as a feature or scenario severity. Scenario inherits feature severity if not provided, or overrides it in the other case. If there is more than one severity definition tag, only the last one is used.

Steps and Scenarious status

Steps with assertion exceptions will be marked as failed. Other exceptions thrown during the test execution will cause it to have status broken. Scenario status will be determined by the first unsuccessful step status. When all steps are passed, then the whole scenario is considered passed.

Step Data

Step data text or table data are represented as step attachments in report.

6.3. Nose

It is a port of pytest-allure-adaptor for nose framework.

6.3.1. Usage

 nosetests --with-allure --logdir=/path/to/put/results
 nosetests --with-allure --logdir=/path/to/put/results --not-clear-logdir

Option "--not-clear-logdir" is useful with option "--processes" to prevent cleaning of logdr at the end of testing.

6.3.2. Supported features

Attachment

To attach some content to test report:

import nose

def test_foo():
    nose.allure.attach('my attach', 'Hello, World')
Step

To divide a test into steps:

import nose

def test_foo():
    with nose.allure.step('step one'):
        # do stuff

    with nose.allure.step('step two'):
        # do more stuff

Can also be used as decorators. By default step name is generated from method name:

import nose

@nose.allure.step
def make_test_data_foo():
    # do stuff

def test_foo():
    assert make_some_data_foo() is not None

@nose.allure.step('make_some_data_foo')
def make_some_data_bar():
    # do another stuff

def test_bar():
    assert make_some_data_bar() is not None
Environment

You can provide test environment parameters such as report name, browser or test server address to allure test report.

import nose

def test_dummy():
    nose.allure.environment(report='Allure report', browser=u'Firefox')
Severity

Any test, class or module can be marked with different severity:

import nose

class TestBar(object):

    @nose.allure.severity(nose.allure.severity_level.CRITICAL)
    def test_bar(self):
        pass

# custom severity
@nose.allure.severity("hard")
def test_bar(self):
    pass

To run tests with concrete priority:

nosetests my_tests/ --with-allure --logdir=tmp --severity="critical, hard"
Issue

Issues can be set for test.

import nose

@nose.allure.issue('http://jira.lan/browse/ISSUE-1')
def test_foo():
    assert False
Features & Stories

Feature and Story can be set for test.

import nose

@nose.allure.feature('Feature1')
@nose.allure.story('Story1')
def test_minor():
    assert False

class TestBar(object):

    @nose.allure.feature('Feature2')
    @nose.allure.story('Story1')
    def test_bar(self):
        pass

To run tests by Feature or Story:

 nosetests my_tests/ --with-allure --logdir=tmp --feature="Feature1, Feature2"
 nosetests my_tests/ --with-allure --logdir=tmp --feature="Feature1, Feature2" --story="Story1, Story2"

7. JavaScript

7.1. Jasmine

A plugin for Allure report generation from Jasmine tests.

7.1.1. Installation

From Jasmine2

Add the lib into package.json and then configure the plugin:

// conf.js
var AllureReporter = require('jasmine-allure-reporter');
jasmine.getEnv().addReporter(new AllureReporter({
  resultsDir: 'allure-results'
}));
From Protractor

Put the above code into the onPrepare inside of your conf.js:

// conf.js
exports.config = {
  framework: 'jasmine2',
  onPrepare: function() {
    var AllureReporter = require('jasmine-allure-reporter');
    jasmine.getEnv().addReporter(new AllureReporter({
      resultsDir: 'allure-results'
    }));
  }
}

7.1.2. Generate HTML report from Allure results

The Reporter will generate xml files inside of a resultsDir, then we need to generate HTML out of them. You can use Maven for that. Copy ready-to-use pom.xml from node_modules/jasmine-allure-reporter and run:

mvn site -Dallure.results_pattern=allure-results

It will put HTMLs into target/site/allure-maven-plugin folder. To serve them via localhost:1324 use:

mvn jetty:run -Djetty.port=1234

Otherwise choose one of the other ways to generate HTML.

7.1.3. Features

This adapter provides a set of methods that can be called from the global Jasmine2AllureReporter object. It wraps common allure JS adapter and provides access to basic Allure features implemented there.

Title
TBD it is not implemented currently
Description

In order to add a detailed description for a test you should call a setDescription(description) method where description is your string argument.

code snippet TBD
Severity

You can specify a severity attribute for a test via severity(severity) method where severity argument can hold one of the following pre-defined values:

  • BLOCKER

  • CRITICAL

  • NORMAL

  • MINOR

  • TRIVIAL

code snippet TBD
Behaviours

In some development approaches tests are classified by stories and features. If you’re using this, then for each test you can set story and feature attributes via story(story) and feature(feature) methods respectively, accepting story or feature as a string argument.

code snippet TBD
Steps

Steps are needed to specify actions that constitute a testing scenario. Steps are named, they can create attachments and can be used in different testing scenarios. You can add a step via createStep(name, stepFunc) method, where

  • name is a string arg bearing step name

  • stepFunc is a wrapped function whose logic this step represents in the report.

code snippet TBD
Attachments

You can add an attachment for one of the steps via createAttachment(name, content, type) method here

  • name is a string bearing attachment’s description

  • content is either a function, returning attachment object or the attachment object itself.

  • type is a string parameter to specify exact MIME type for each attached file. However there’s no need to specify attachment explicitly type for all attached files as Allure by default analyses attachment contents and can determine attachment type automatically. You usually need to specify attachment type when working with plain text files.

Example: adding a screenshot in the end of each test

  onPrepare: function () {
    var AllureReporter = require('jasmine-allure-reporter');
    jasmine.getEnv().addReporter(new AllureReporter());
    jasmine.getEnv().afterEach(function(done){
      browser.takeScreenshot().then(function (png) {
        allure.createAttachment('Screenshot', function () {
          return new Buffer(png, 'base64')
        }, 'image/png')();
        done();
      })
    });
  }

Note done callback!

Issues Tracker
Integration with bug tracker system is not currently implemented
Test Management System
Integration with test management system is not currently implemented
Parameters

In order to add information about test method parameters you should use one of the methods:

  1. addArgument(name, value) - to specify more information on one of the test arguments

  2. addEnvironment(name, value) - to specify more information on some of the environment variables

code snippet TBD

7.1.4. TBD

  • Currently attachments are added to the test case instead of the current step. This needs to be fixed in allure-js-commons.

  • Add support for Features.

  • Add support to Jasmine1. Right now only Jasmine2 is available (do we really need this?).

  • Add ability to use reflection for decoration method of page objects so that we don’t need to write Allure-related boilerplate tying ourselves to one specific reporter.

7.1.5. For Developers

See the system tests to quickly check how the reporter works in real life:

node_modules/protractor/bin/protractor ./test/system/conf.js

7.2. Cucumber JS

Allure report version supported : 1.4.15

7.2.1. Usage:

Add reporter.js file in feature packages with:

var reporter = require('cucumberjs-allure-reporter');
reporter.config(
    {...}
);
module.exports = reporter;

Supported configuration keys:

  • targetDir - directory where allure will save results xml’s

  • labels - custom label matchers, example:

labels : {
        feature: [/sprint-(\d*)/, /release-(\d)/, /area-(.*)/],
        issue: [/(bug-\d*)/]
    }

Possible labels:

  • feature

  • story

  • severity

  • language

  • framework

  • issue

  • testId

  • host

  • thread

If you want to cancel step or test, simply throw new Error with message 'Step cancelled' or 'Test cancelled'.

7.2.2. Generate HTML report from Allure results

The Reporter will generate xml files inside of a targetDir, then we need to generate HTML out of them. You can use Maven for that. Copy ready-to-use pom.xml from node_modules/cucumberjs-allure-reporter and run:

mvn site -Dallure.results_pattern=allure-results

It will put HTMLs into target/site/allure-maven-plugin folder. To serve them via localhost:1234 use:

mvn jetty:run -Djetty.port=1234

Otherwise choose one of other ways to generate HTML.

7.2.3. For Developers

Run test examples with:

./node_modules/.bin/cucumber.js features/<FEATURE_NAME>.feature

Available tests:

  • basic → basic test results

  • description → scenario description test

  • label → cucumber tags (currently labels are not visible in generated report …​)

  • exception → test throws exception

  • attachments → docStrings and dataTable tests

  • scenarioOutline → scenario outline tests

  • subSteps → steps add sub steps using allure object

or run everything with: ./node_modules/.bin/cucumber.js features/

To check protractor screenshot test install protractor and protractor-cucumber-framework and then run tests: ./node_modules/protractor/bin/protractor miscellaneous/protractorScreenshot/conf.js

To check basic logging run: ./node_modules/.bin/cucumber.js miscellaneous/logging

To check basic configuration run: ./node_modules/.bin/cucumber.js miscellaneous/configuration

To check custom tags run: ./node_modules/.bin/cucumber.js miscellaneous/customTagNames

7.2.4. Release notes

01/09/2016 version 1.0.3

  • added possibility to cancel steps and tests

11/07/2016 version 1.0.2

  • peer dependency for cucumber (>= 1.2.0) added

06/07/2016 version 1.0.1

  • dependencies updated for allure-js-commons, protractor, cucumber and protractor-cucumber-framework

  • fixed cucumber handlers - getPayload will not be available in upcoming cucumber major release

  • updated data table and doc string handling

02/12/2015 version 1.0.0

  • plugin updated to work with cucumber 0.9.1

01/09/2015 version 0.0.1

  • first release

7.3. Karma

Reporter for the Allure XML format. It allows to make a detailed report

7.3.1. karma-allure-reporter

Install karma-allure-reporter into your project as devDependency

{
  "devDependencies": {
    "karma": "~0.10",
    "karma-allure-reporter": "~1.0.0"
  }
}

You can simple do it by:

npm install karma-allure-reporter --save-dev

7.3.2. Configuration

Add allure into reporters section. Allure-reporter has a single config, it’s a reportDir — result files location relatively to base dir. By default, files save right in the base dir.

// karma.conf.js
module.exports = function(config) {
  config.set({
    reporters: ['progress', 'allure'],

    // the default configuration
    allureReport: {
      reportDir: '',
    }
  });
};

You can pass list of reporters as a CLI argument too:

karma start --reporters allure,dots

7.3.3. API

With allure reporter you get some functions for provide additional info about tests. All functions available as methods of the global allure object.

  • description(description) assign a description to current testcase

  • severity(severity) assign a severity to current testcase. Possible values enumerated as properties, eg. allure.severity.BLOCKER. All securities by descending of their importance:

    • BLOCKER

    • CRITICAL

    • NORMAL

    • MINOR

    • TRIVIAL

  • createStep(name, stepFunction) defines test step. Returns wrapped function which reports about every step calling. Step function can be nested within one another. It is most powerful feature of allure, because it allow to write self-documented tests which report about every its step.

See the docs in core project for more information about these features and their purpose.

7.3.4. Example

There is an example project, where you may look to allure-reporter in action.

For more information about Allure see the allure core project.

For more information about Karma see the Karma homepage.

7.4. Mocha

7.4.1. mocha-allure-reporter

Allure reporter for Mocha framework.

7.4.2. Installation

Assuming that you have mocha installed, install reporter via npm:

npm install mocha-allure-reporter

Then use it as any other mocha reporter:

mocha --reporter mocha-allure-reporter

After running tests you will get raw tests result into allure-results directory. See generator list on how to make a report from raw results.

Also check out mocha-allure-example to see it in action.

7.4.3. Supported options

  • targetDir (string) – directory where test results will be stored

7.4.4. Runtime API

Allure is a test framework which provides more data from tests than usual. Once added mocha-allure-reporter will create global allure object with the following API:

  • allure.createStep(name, stepFn) – define step function. Result of each call of this function will be recorded into report.

  • allure.createAttachement(name, content, [type]) – save attachment to test. If you’re calling this inside step function or during its execution (e.g. asynchronously via promises), attachment will be saved to step function.

    • name (String) - attachment name. Note that it is not then name of the file, actual filename will be generated.

    • content (Buffer|String|Function) – attachment content. If you pass Buffer or String, it will be saved to file immediately. If you are passing Function, you will get decorated function and you can call it several times to trigger attachment. General purpose of the second case is an ability to create utility function to take screenshot. You can define function for you test framework only once and then call it each time you need a screenshot.

    • type (String, optional) – attachment MIME-type. If you omit this argument we’ll try to detect type automatically via file-type library

  • allure.description(description) – set detailed test description, if test name is not enough.

  • allure.severity(severity) – set test severity, one of: blocker, critical, normal, minor, trivial. You can also use constants like allure.SEVERITY.BLOKER.

  • allure.feature(featureName) – assign feature to test

  • allure.story(storyName) – assign user story to test. See documentation for details

  • allure.addArgument(name, value) - provide parameters, which had been used in test. Unlike other languages, javascript test methods usually doesn’t have special arguments (only callbacks), so developers use other way to populate parameters to test. This method is to provide them to Allure

  • allure.addEnvironment(name, value) - save environment value. It is similar to addArgument method, but it is designed to store more verbose data, like HTTP-links to test page or used package version.

8. Ruby

8.1. Cucumber

This page describes Allure adaptor usage for Cucumber framework.

8.1.1. Installation

Add this line to your application’s Gemfile:

gem 'allure-cucumber'

And then execute:

$ bundle

Or install it yourself as:

$ gem install allure-cucumber

8.1.2. Configuration

By default, Allure XML files are stored in gen/allure-results. To change this add the following in features/support/env.rb file:

AllureCucumber.configure do |c|
   c.output_dir = "/output/dir"
end

By default, the Allure XML files from earlier runs are deleted at the start of new run. This can be disabled by:

AllureCucumber.configure do |c|
  c.clean_dir  = false
end

By default, allure-cucumber will analyze your cucumber tags looking for Test Management, Issue Management, and Severity hooks. These hooks will be displayed in the generated allure report (see allure-core for further info).

    DEFAULT_TMS_PREFIX      = '@TMS:'
    DEFAULT_ISSUE_PREFIX    = '@ISSUE:'
    DEFAULT_SEVERITY_PREFIX = '@SEVERITY:'

Example:

  @SEVERITY:trivial @ISSUE:YZZ-100 @TMS:9901
  Scenario: Leave First Name Blank
    When I register an account without a first name
    Then exactly (1) [validation_error] should be visible

You can configure what allure-cucumber looks for by making the following changes

AllureCucumber.configure do |c|
  c.clean_dir  = false
  c.tms_prefix      = '@HIPTEST--'
  c.issue_prefix    = '@JIRA++'
  c.severity_prefix = '@URGENCY:'
end

Example:

  @URGENCY:critical @JIRA++YZZ-100 @HIPTEST--9901
  Scenario: Leave First Name Blank
    When I register an account without a first name
    Then exactly (1) [validation_error] should be visible

8.1.3. Usage

Put the following in your features/support/env.rb file:

require 'allure-cucumber'

Use --format AllureCucumber::Formatter --out where/you-want-results while running cucumber or add it to cucumber.yml

You can also attach screenshots, logs or test data to steps.

 #file: features/support/env.rb
 include AllureCucumber::DSL
 attach_file(title, file)

8.1.4. How to generate report

This adapter only generates XML files containing information about tests. See wiki section on how to generate report.

8.2. RSpec

Adaptor to use the Allure framework along with the RSpec. See an example project to take a quick tour.

8.2.1. What’s new

See the releases tab.

8.2.2. Setup

Add the dependency to your Gemfile. Choose the version carefully:

  • 0.5.x - for RSpec2.

  • ⇐ 0.6.7 - for RSpec < 3.2.

  • >= 0.6.9 - for RSpec >= 3.2.

 gem 'allure-rspec'

And then include it in your spec_helper.rb:

    RSpec.configure do |c|
      c.include AllureRSpec::Adaptor
    end

8.2.3. Advanced options

You can specify the directory where the Allure test results will appear. By default it would be 'gen/allure-results' within your current directory.

    AllureRSpec.configure do |c|
      c.output_dir = "/whatever/you/like" # default: gen/allure-results
      c.clean_dir = false # clean the output directory first? (default: true)
      c.logging_level = Logger::DEBUG # logging level (default: DEBUG)
    end

8.2.4. Usage examples

describe MySpec, :feature => "Some feature", :severity => :normal do

  before(:step) do |s|
    puts "Before step #{s.current_step}"
  end

  it "should be critical", :story => "First story", :severity => :critical, :testId => 99 do
    "string".should == "string"
  end

  it "should be steps enabled", :story => ["First story", "Second story"], :testId => 31 do |e|

    e.step "step1" do |s|
      s.attach_file "screenshot1", take_screenshot_as_file
    end

    e.step "step2" do
      5.should be > 0
    end

    e.step "step3" do
      0.should == 0
    end

    e.attach_file "screenshot2", take_screenshot_as_file
  end
end

9. Groovy

9.1. Spock

Spock Framework sources - https://github.com/spockframework/spock

9.1.1. Spock Framework compatibilities

10. PHP

10.1. PHPUnit

This an official PHPUnit adapter for Allure Framework - a flexible, lightweight and multi-language framework for writing self-documenting tests.

10.1.1. What is this for?

The main purpose of this adapter is to accumulate information about your tests and write it out to a set of XML files: one for each test class. Then you can use a standalone command line tool or a plugin for popular continuous integration systems to generate an HTML page showing your tests in a good form.

10.1.2. Example project

10.1.3. How to generate report

This adapter only generates XML files containing information about tests. See wiki section on how to generate report.

10.1.4. Installation && Usage

Note: this adapter supports Allure 1.4.x only. In order to use this adapter you need to add a new dependency to your composer.json file:

{
    "require": {
        "php": ">=5.4.0",
        "allure-framework/allure-phpunit": "~1.2.0"
    }
}

Then add Allure test listener in phpunit.xml file:

<listeners>
    <listener class="Yandex\Allure\Adapter\AllureAdapter" file="vendor/allure-framework/allure-phpunit/src/Yandex/Allure/Adapter/AllureAdapter.php">
        <arguments>
            <string>build/allure-results</string> <!-- XML files output directory -->
            <boolean>true</boolean> <!-- Whether to delete previous results on rerun -->
            <array> <!-- A list of custom annotations to ignore (optional) -->
                <element key="0">
                    <string>someCustomAnnotation</string>
                </element>
            </array>
        </arguments>
    </listener>
</listeners>

After running PHPUnit tests a new folder will be created (build/allure-results in the example above). This folder will contain generated XML files. See framework help for details about how to generate report from XML files. By default generated report will only show a limited set of information but you can use cool Allure features by adding a minimum of test code changes. Read next section for details.

10.1.5. Main features

This adapter comes with a set of PHP annotations and traits allowing to use main Allure features.

Human-readable test class or test method title

In order to add such title to any test class or test case method you need to annotate it with @Title annotation:

namespace Example\Tests;

use PHPUnit_Framework_TestCase;
use Yandex\Allure\Adapter\Annotation\Title;

/**
 * @Title("Human-readable test class title")
 */
class SomeTest extends PHPUnit_Framework_TestCase
{
    /**
     * @Title("Human-readable test method title")
     */
    public function testCase()
    {
        //Some implementation here...
    }
}
Extended test class or test method description

Similarly you can add detailed description for each test class and test method. To add such description simply use @Description annotation:

namespace Example\Tests;

use PHPUnit_Framework_TestCase;
use Yandex\Allure\Adapter\Annotation\Description;
use Yandex\Allure\Adapter\Model\DescriptionType;

/**
 * @Description("Detailed description for test class")
 */
class SomeTest extends PHPUnit_Framework_TestCase
{
    /**
     * @Description(value = "Detailed description for <b>test class</b>.", type = DescriptionType::HTML)
     */
    public function testCase()
    {
        //Some implementation here...
    }
}

Description can be added in plain text, HTML or Markdown format - simply assign different type value.

Set test severity

@Severity annotation is used in order to prioritize test methods by severity:

namespace Example\Tests;

use PHPUnit_Framework_TestCase;
use Yandex\Allure\Adapter\Annotation\Severity;
use Yandex\Allure\Adapter\Model\SeverityLevel;

class SomeTest extends PHPUnit_Framework_TestCase
{
    /**
     * @Severity(level = SeverityLevel::MINOR)
     */
    public function testCase()
    {
        //Some implementation here...
    }
}
Specify test parameters information

In order to add information about test method parameters you should use @Parameter annotation:

namespace Example\Tests;

use PHPUnit_Framework_TestCase;
use Yandex\Allure\Adapter\Annotation\Parameter;
use Yandex\Allure\Adapter\Model\ParameterKind;

class SomeTest extends PHPUnit_Framework_TestCase
{
    /**
     * @Parameter(name = "param1", value = "value1")
     * @Parameter(name = "param2", value = "value2", kind = ParameterKind::SYSTEM_PROPERTY)
     */
    public function testCase()
    {
        //Some implementation here...
    }
}
Map test classes and test methods to features and stories

In some development approaches tests are classified by stories and features. If you’re using this then you can annotate your test with @Stories and @Features annotations:

namespace Example\Tests;

use PHPUnit_Framework_TestCase;
use Yandex\Allure\Adapter\Annotation\Features;
use Yandex\Allure\Adapter\Annotation\Stories;

/**
 * @Stories({"story1", "story2"})
 * @Features({"feature1", "feature2", "feature3"})
 */
class SomeTest extends PHPUnit_Framework_TestCase
{
    /**
     * @Features({"feature2"})
     * @Stories({"story1"})
     */
    public function testCase()
    {
        //Some implementation here...
    }
}

You will then be able to filter tests by specified features and stories in generated Allure report.

Attach files to report

If you wish to attach some files generated during PHPUnit run (screenshots, log files, dumps and so on) to report - then you need to use AttachmentSupport trait in your test class:

namespace Example\Tests;

use PHPUnit_Framework_TestCase;
use Yandex\Allure\Adapter\Support\AttachmentSupport;

class SomeTest extends PHPUnit_Framework_TestCase
{

    use AttachmentSupport;

    public function testCase()
    {
        //Some implementation here...
        $filePath = $this->outputSomeContentToTemporaryFile();
        $this->addAttachment($filePath, 'Attachment human-readable name', 'text/plain');
        //Some implementation here...
    }

    private function outputSomeContentToTemporaryFile()
    {
        $tmpPath = tempnam(sys_get_temp_dir(), 'test');
        file_put_contents($tmpPath, 'Some content to be outputted to temporary file.');
        return $tmpPath;
    }

}

In order to create an attachment simply call AttachmentSupport::addAttachment() method. This method accepts attachment type, human-readable name and a string either storing full path to the file we need to attach or file contents.

Divide test methods into steps

Allure framework also supports very useful feature called steps. Consider a test method which has complex logic inside and several assertions. When an exception is thrown or one of assertions fails sometimes it’s very difficult to determine which one caused the failure. Allure steps allow to divide test method logic into several isolated pieces having independent run statuses such as passed or failed. This allows to have much more cleaner understanding of what really happens. In order to use steps simply import StepSupport trait in your test class:

namespace Example\Tests;

use PHPUnit_Framework_TestCase;
use Yandex\Allure\Adapter\Support\StepSupport;

class SomeTest extends PHPUnit_Framework_TestCase
{

    use StepSupport;

    public function testCase()
    {
        //Some implementation here...
        $this->executeStep("This is step one", function () {
            $this->stepOne();
        });

        $this->executeStep("This is step two", function () {
            $this-stepTwo();
        });

        $this->executeStep("This is step three", function () {
            $this->stepThree('someArgument');
        });
        //Some implementation here...
    }

    private function stepOne()
    {
        //Some implementation here...
    }

    private function stepTwo()
    {
        //Some implementation here...
    }

    private function stepThree($argument)
    {
        //Some implementation here...
    }

}

The entire test method execution status will depend on every step but information about steps status will be stored separately.

10.2. ALLURECodeception

This is an official Codeception adapter for Allure Framework.

10.2.1. What is this for?

The main purpose of this adapter is to accumulate information about your tests and write it out to a set of XML files: one for each test class. This adapter only generates XML files containing information about tests. See wiki section on how to generate report.

10.2.2. Example project

10.2.3. Installation and Usage

In order to use this adapter you need to add a new dependency to your composer.json file:

{
    "require": {
        "php": ">=5.4.0",
        "allure-framework/allure-codeception": "~1.1.0"
    }
}

To enable this adapter in Codeception tests simply put it in "enabled" extensions section of codeception.yml:

extensions:
    enabled:
        - Yandex\Allure\Adapter\AllureAdapter
    config:
        Yandex\Allure\Adapter\AllureAdapter:
            deletePreviousResults: false
            outputDirectory: allure-results

deletePreviousResults will clear all .xml files from output directory (this behavior may change to complete cleanup later). It is set to false by default.

outputDirectory is used to store Allure results and will be calculated relatively to Codeception output directory (also known as paths: log in codeception.yml) unless you specify an absolute path. You can traverse up using .. as usual. outputDirectory defaults to allure-results.

To generate report from your favourite terminal, install allure-cli and run following command (assuming you’re in project root and using default configuration):

allure generate --report-version 1.4.5 --report-path tests/_output/allure-report -- tests/_output/allure-results

Report will be generated in tests/_output/allure-report.

10.2.4. Main features

See respective PHPUnit section.

11. .Net

11.1. MSTest

MSTestAllureAdapter allows you convert an MSTest trx file to the XMLs from which an Allure report can be generated.

Because MSTest does not have an extension or hook mechanism this adapter cannot run as part of MSTest but instead it converts the resulting trx file to the xml format expected by allure.

It is a .NET/Mono based console application.

11.1.1. Usage

MSTestAllureAdapter.Console.exe <TRX file> [output target dir]

If '[output target dir]' is missing the reslts are saved in the current directory in a folder named 'results'.

$ mono MSTestAllureAdapter.Console.exe sample.trx

This will generate the xml files from which the allure report can be created based upon the 'sample.trx' file.

$ mono MSTestAllureAdapter.Console.exe sample.trx output-xmls

This will generate the xml files from which the allure report can be created in a folder named 'output-xmls' based upon the 'sample.trx' file.

If the target directory does not exists it is created.

To generate a report using allure-cli:

$ allure generate output-xmls -v 1.4.0

11.2. NUnit

11.2.1. Installation and Usage

  1. Download NUnit 2.6.3 or 2.6.4 from official site;

  2. Download latest release from allure-nunit2 releases for corresponding version (version specified in parentheses);

  3. Unpack binaries to %NUnit_installation_directory%\bin\addins;

  4. NOTE: addin will NOT be visible in Tools → Addins.. because it’s built against .NET 4.0;

  5. In %NUnit_installation_directory%\bin\addins\config.xml specify ABSOLUTE path to any folder (this folder will be created automatically) where xml files will be generated (e.g. <results-path>C:\test-results\AllureResults</results-path> or <results-path>/home/user/test-results/AllureResults</results-path>);

  6. You can also specify in configuration whether you want to take screenshots after failed tests and whether you want to have test output to be written to attachments.

  7. Run your tests with NUnit GUI or nunit-console using .NET 4.0 (e.g. nunit-console YourAssembly.dll /framework=net-4.0);

  8. After all tests finish you’ll see new folder that you specified on step 5 with generated xml files;

11.2.2. How to generate report

This adapter only generates XML files containing information about tests. See wiki section on how to generate report.

11.2.3. What about NUnit 3?

It’s still in alpha-stage, too early to develop an adapter for it. Adapter for NUnit 3 will be located in it’s own repository.

11.2.4. Further reading

12. Scala

12.1. ScalaTest

This adapter allows to retrieve test execution data from ScalaTest framework and convert it to the form suitable for Allure report generation.

12.1.1. Example project

12.1.2. Installation, Configuration and Usage

In order to use this adapter you need to have JDK 1.7+ installed. To enabled adapter simply add the following dependency to build.sbt:

libraryDependencies += "ru.yandex.qatools.allure" % "allure-scalatest_2.10" % "1.4.0-SNAPSHOT"

Then attach AllureReporter in build.sbt:

testOptions in Test ++= Seq(
    Tests.Argument(TestFrameworks.ScalaTest, "-oD"),
    Tests.Argument(TestFrameworks.ScalaTest, "-C", "ru.yandex.qatools.allure.scalatest.AllureReporter")
)

When using this adapter you can encounter the following warning:

[ScalaTest-dispatcher] WARN ru.yandex.qatools.allure.utils.AllureResultsUtils - Can't set "com.sun.xml.bind.marshaller.CharacterEscapeHandler" property to given marshaller
javax.xml.bind.PropertyException: name: com.sun.xml.bind.marshaller.CharacterEscapeHandler value: ru.yandex.qatools.allure.utils.BadXmlCharacterEscapeHandler@5e652b7b
    at javax.xml.bind.helpers.AbstractMarshallerImpl.setProperty(AbstractMarshallerImpl.java:358)
    at com.sun.xml.internal.bind.v2.runtime.MarshallerImpl.setProperty(MarshallerImpl.java:527)
    at ru.yandex.qatools.allure.utils.AllureResultsUtils.setPropertySafely(AllureResultsUtils.java:199)
    at ru.yandex.qatools.allure.utils.AllureResultsUtils.marshaller(AllureResultsUtils.java:171)
    at ru.yandex.qatools.allure.utils.AllureResultsUtils.writeTestSuiteResult(AllureResultsUtils.java:148)
    at ru.yandex.qatools.allure.Allure.fire(Allure.java:180)
    at ru.yandex.qatools.allure.scalatest.AllureReporter.testSuiteFinished(AllureReporter.scala:74)
    at ru.yandex.qatools.allure.scalatest.AllureReporter.apply(AllureReporter.scala:46)
    at org.scalatest.DispatchReporter$Propagator$$anonfun$run$1.apply(DispatchReporter.scala:240)
    at org.scalatest.DispatchReporter$Propagator$$anonfun$run$1.apply(DispatchReporter.scala:239)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at org.scalatest.DispatchReporter$Propagator.run(DispatchReporter.scala:239)
    at java.lang.Thread.run(Thread.java:744)

This is related to incompatible JAXB versions used in Allure and Scalatest so you can safely ignore it.

12.1.3. How to generate report

This adapter only generates XML files containing information about tests. See [wiki section] (https://github.com/allure-framework/allure-core/wiki#generating-report) on how to generate report.

12.1.4. Publishing to Sonatype (adapter development)

A publicly available (on public keyserver) GPG key should be present in you default GPG keyring. You need to create sonatype.sbt file in ~/.sbt/<sbt-version>/:

credentials += Credentials("Sonatype Nexus Repository Manager",
                           "oss.sonatype.org",
                           "login",
                           "password")

To publish simply run:

$ sbt publish-signed

12.2. Specs

This adapter allows to retrieve test execution data from Specs framework and convert it to the form suitable for Allure report generation.

12.2.1. Example project

12.2.2. Usage

In order to use this adapter you need to have JDK 1.7+ installed. To enabled adapter simply add the following dependency to build.sbt:

libraryDependencies += "ru.yandex.qatools.allure" % "allure-specs2_2.10" % "1.4.0-SNAPSHOT"

Then attach AllureReporter in build.sbt:

testOptions in Test ++= Seq(
  Tests.Argument(TestFrameworks.Specs2, "notifier", "ru.yandex.qatools.allure.specs2.AllureNotifier")
)

13. Reporting

Once the XML files have been created by the adapter, you can generate an Allure report using one of the following tools

13.1. Commandline

Using Allure command line tool you can generate the report from existing Allure results.

13.1.1. Installation

Allure CLI is a Java application so it’s available for all platforms. You have to manually install Java 1.8 before using Allure CLI

Mac OS

You can install Allure CLI via Homebrew.

$ brew tap qameta/allure
$ brew install allure

After installation you will have allure command available. Read more about Allure Homebrew Formula.

Debian

For Debian-based repositories we provide a PPA so the installation is straightforward:

$ sudo apt-add-repository ppa:yandex-qatools/allure-framework
$ sudo apt-get update
$ sudo apt-get install allure-commandline

Supported distributions are: Trusty and Precise. After installation you will have allure command available. Read more about Allure Debian Package.

Windows and other Unix
  • Download the latest version as zip archive from bintray.

  • Unpack the archive to allure-commandline directory.

  • Navigate to bin directory.

  • Use allure.bat for Windows and allure for other Unix platforms.

  • Add allure to system PATH.

13.1.2. Configuration

Properties file

To configure the Allure report you should create allure.properties file

allure.properties
allure.issues.tracker.pattern=https://tms/testcase-%s
allure.tests.management.pattern=https://bugtracker/issue-%s

and simply generate the report. By default command line tool looking for config in directory you run the command. You can use ALLURE_CONFIG environment variable to specify the path to configuration.

Command line arguments

There are an options you can use directly from command line. To show them, execute the command

$ allure help
Usage

There is few commands available.

Generate the report

To generate the report from existing Allure results you can use the following command:

$ allure generate directory-with-results/

The report will be generated to allure-report folder. You can change the destination folder using -o flag:

$ allure generate directory-with-results/ -o directory-with-report

For more information use the allure help generate command.

Open the report

When the report is generated you can open it in your default system browser. Simply run

$ allure report open
Clean the report

If you want to remove the generated report data use allure report clean command.

By default the report commands will looking for the report in allure-results folder. If you are want to use the report from different location you can use -o option.

13.2. Jenkins

13.2.1. Installation

First part of installation guide concerns global installation of Allure plugin and it’s commandline tool. Note that unlike to previous versions Jenkins Allure Plugin since 2.11 does not require to configure Jenkins Content Security Policy properties.

  1. Install the latest version of Allure Plugin from "Plugin Manager" page.

  2. Open Plugin Advanced Settings (at <your_jenkins_host>/pluginManager/advanced).

  3. Click "Check Updates" button on the bottom right corner of the page.

  4. Configure Allure Commandline from "Global Tools Configuration".

Build agents have connection to Maven repository with Allure distribution
  1. Open "Global Tool Configuration".

  2. Find "Allure Commandline" configuration block.

    Find Allure Commandline
  3. Click "Add Allure Commandline" button.

    Install Allure Commandline
  4. Fill the name and choose the version in the "From Maven Central" dropdown.

Downloading and installing Allure from archive
  1. Go to the Bintray page, click the latest released version, and download the allure-[version].zip file

  2. Upload downloaded file to any file server. Build agents must have access to the file by url.

  3. Open "Global Tool Configuration". and find "Allure Commandline" configuration block.

  4. Click "Add Allure Commandline" button.

    Install Allure Commandline
  5. Click "Add Installer" and choose "Extract .zip/.tar.gz".

    Install Allure Commandline
  6. Fill the name and download url. Mind that Jenkins will check the provided url immediately and show a warning if it’s not available.

    Install Allure Commanline

13.2.2. Configuration

Job configuration
  1. Open job configuration page.

  2. Ensure that your build generates some data in one of the formats digestible for Allure Commandline tool.

  3. Add Allure Report post build action.

    Select
  4. Configure the Allure Report.

    install allure commanline
Test Management and Bug-Tracking systems integrations

Currently all versions of Allure support setting of url patterns to make direct links to your TMS or bug-tracker available from the report. (Learn more about this integrations link-[here]) It may be done by setting special system properties:

  1. allure.tests.management.pattern property:

    Filling TMS link property
  2. allure.issues.tracker.pattern property:

    Filling bug-tracker link property

in both patterns %s will be substituted for a string value obtained from the label added to test case.

Advanced Options

"Advanced…​" button provides you with several additional settings:

  • jdk to use

  • build policy value

  • include environment flag - currently is not implemented

Allure build step advanced options

13.2.3. Usage

When build is finished a link to Allure report will appear on the build page and also in the build history entries:

Install Allure Commanline

Generated Allure report is stored in the artifacts folder of the build and can be processed the same way as all the other artifacts, or downloaded as .zip archive.

install allure commanline

13.2.4. History files

Allure since version 2 supports history for tests in the report [Learn more about History plugin]. At the every report generation during the build Jenkins Allure Plugin will try to access working directory of the previous build and copy contents of allure-report\history folder to the current report contents. At the moment history entry for test case stores information about up to 5 previous results.

13.2.5. Extensions

Job DSL Plugin

The job-dsl-plugin allows the programmatic creation of projects using a DSL.

// default
publishers {
    allure(['allure-results'])
}

// advanced
publishers {
    allure(['first-results', 'second-results']) {
        jdk('java7')
        commandline('1.4.18')

        buildFor('UNSTABLE')
        includeProperties(true)
        property('allure.issues.tracker.pattern', 'http://tracker.company.com/%s')
        property('allure.tests.management.pattern', 'http://tms.company.com/%s')
    }
}
Pipeline Plugin

To add Allure report generation from pipeline steps one can use Pipeline Syntax builder to generate Allure step code. Note, that commandline parameter points to Allure Commandline installation name to use, if it is not provided, plugin will pick first suitable installation found automatically.

Pipeline step generation

which gives a pipeline script:

node {
// script body

allure([
         includeProperties: false,
         jdk: '',
         properties: [[key: 'allure.issues.tracker.pattern', value: 'http://tracker.company.com/%s']],
         reportBuildPolicy: 'ALWAYS',
         results: [[path: 'target/allure-results'], [path: 'other_target/allure-results']]
         ])
}

To use Allure Report from the pipeline steps one can invoke Allure DSL method as script:

stage('reports') {
    steps {
    script {
            allure([
                    includeProperties: false,
                    jdk: '',
                    properties: [],
                    reportBuildPolicy: 'ALWAYS',
                    results: [[path: 'target/allure-results']]
            ])
    }
    }
}

13.3. TeamCity

13.3.1. Installation

Install Allure TeamCity plugin. Copy the allure.zip from (latest release to the <TeamCity Data Directory>/plugins/.tools directory. No server restart needed for this step.

13.3.2. Configuration

  1. Open the build configuration settings.

  2. Ensure that your build generates Allure XML files.

  3. Go to Build steps and add the Allure Report build step.

    TeamCity Plugin Install
  4. Configure the step.

    jenkins plugin install

In case you upgrading the Allure TeamCity plugin you need to remove old Allure report generation feature.

13.3.3. Usage

When the build is done you will get Allure Report as a part of build artifacts – simply open the index.html. image::teamcity_plugin_usage.png[jenkins plugin install]

13.4. Bamboo

13.4.1. Installation

  1. Install Allure Plugin from "Plugin Manager Page".

  2. Download the latest version of allure-commandline.zip from github and extract to the desired directory.

  3. Defining a new executable capability with using the extraction directory.

13.4.2. Configuration

  1. Open the build configuration settings.

  2. Ensure that your build generates Allure XML files.

  3. Add Allure task to your job.

    Add Task in Bamboo
  4. Configure task - specify glob pattern to the folder where Allure should search for XML files.

    Task configuration

13.4.3. Usage

When the build is done you will get Allure Report as a part of build artifacts. Сlick on Allure report artifact on the Artifacts tab:

Allure artifact

13.5. Gradle

Allure plugin for Gradle allows integration of Allure listeners into TestNG, Junit4, Cucumber JVM and Spock frameworks in your Gradle-based projects, and also provides several tasks for building a single-project or multi-project report.

13.5.1. Autoconfiguration for Junit4 and TestNG

Plugin is able to use Gradle integration for Junit4 and TestNG, so in case you are using one of this frameworks, configuration will look very simple:

buildscript {

    repositories {
        jcenter()
    }

    dependencies {
        classpath "io.qameta.allure:allure-gradle:2.3"
    }
}

    plugins {
        id 'io.qameta.allure'
    }

    allure {
        version = '2.2.1'
        autoconfigure = true
     }

Here you will need only two parameters: autoconfigure - it is a flag that enables framework detection. If you enable it, plugin will automatically add the right listener and a dependency for aspectjweaver runtime agent and version - parameter that specifies the Allure Commandline to download and build a report with.

After you apply automatic configuration, Allure results obtained after the test task execution will be stored in build/allure-results.

To build a report execute the allure Gradle task:

./gradlew clean build allure

Note, that if you don’t specify the version, plugin will not create any Allure-related tasks. So if you only want to create results for further processing, omit the version parameter.

13.5.2. Configuration for a specific testing framework

Plugin provides a configuration closure parameter for 4 different frameworks: Junit4, TestNG, Cucumber JVM and Spock.

Let’s see how the Spock-based build script will look:

buildscript {

    repositories {
        jcenter()
    }

    dependencies {
        classpath "io.qameta.allure:allure-gradle:2.3"
    }
}

plugins {
        id 'io.qameta.allure'
}

allure {
    version = '2.2.1'
    aspectjweaver = true

    useSpock {
        version = '2.0-BETA12'
    }
}

Here aspectjweaver flag is used to add a corresponding dependency for runtime agent to your project, and useSpock configures a version for Allure Spock adaptor dependency.

13.5.3. Full configuration

allure {
    version = '2.2.1'

    autoconfigure = false

    String allureJavaVersion = '2.0-BETA9'

    aspectjweaver = true

    boolean clean = true

    resultsDir = file('/path/to/project/module/build/allure-results')

    reportDir = file('build/reports/allure-report')

    String configuration = 'testCompile'

    useJunit4 {
       version = '2.0-BETA12'
    }

    useTestNG {
       version = '2.0-BETA12'
       spi-off = false
    }

    useCucumberJVM {
       version = '2.0-BETA12'
    }

    useSpock {
       version = '2.0-BETA12'
    }

    downloadLink = 'https://dl.bintray.com/qameta/generic/io/qameta/allure/allure/2.2.1/allure-2.2.1.zip'
}
  • boolean autoconfigure (default = false) - a flag to specify usage of autoconfiguration, plugin will attempt to find test platform integration provided by Gradle (currently works only for Junit4 and TestNG).

  • boolean aspectjveaver - a flag to specify inclusion/exclusion of aspectjweaver runtime agent

  • boolean clean (default = true) - enable --clean option for the Allure commandline tool, true by default.

  • String version - specify report generator version, note, this property is necessary to enable allure and aggregatedAllureReport tasks

  • String configuration (default = 'testCompile') - configuration name where to bind plugin dependencies

  • File resultsDir - directory for Allure results in the current project, build\allure-results by default

  • File reportDir - directory for Allure report in the current project, build\reports\allure-report by default

  • String allureJavaVersion (default = '2.0-BETA9') - version of allure java release to be used for autoconfiguration

  • String downloadLink - custom location of Allure distribution to download from, by default allure is downloaded from bintray by specified version and installed into .allure folder in the project root.

13.5.4. Tasks

You can use some of the tasks that are defined in the plugin for your own convenience.

allure

Creates Allure report for a given list of input directories

Parameters:

  • File reportDir - destination directory for Allure report

  • boolean clean - enable --clean option for the Allure commandline tool

  • String version - Allure Commandline version, will attempt to discover an installation of commandline with this version in the .allure folder in the root of your project.

  • List<File> resultsDirs - list of directories with Allure results.

So to generate an Allure report for a multi-module project you will have to define your own allureAggregatedReport task, for example:

plugins {
    id 'io.qameta.allure'
}

allprojects { project ->

    apply plugin: 'io.qameta.allure'

    allure {
        version = '2.1.0'
    }
}

import io.qameta.allure.gradle.task.AllureReport

task allureAggregatedReport(type: AllureReport) {
    resultsDirs = subprojects.allure.resultsDir
}

And invoke it like: ./gradlew clean build allureAggregatedReport

downloadAllure

Downloads Allure Commandline from provided url and saves it into .allure folder in your project’s root directory.

  • String src - url specifying download location for Allure Commandline.

  • String version - version parameter is needed to find the right version of Allure in the downloaded archive.

  • File dest - destination folder to install downloaded Allure Commandline.

allure task depends on downloadAllure, so in the default case there is no need to invoke it manually, whenever allure task is invoked, it will call downloadAllure first.

serve

Can be used to generate and open report in the default browser after the build.

  • String version - version to discover Allure Commandline installation in the .allure/version

  • List<File> resultsDirs - list of folders with results.

By default serve task is configured by plugin for a single-module report and when executed will open it in the default browser.

usage example: ./gradlew clean build serve

13.6. Maven

This plugin generates Allure report by existing XML files during the Maven build process.

13.6.1. Installation

Add allure-maven-plugin to your pom.xml file build section.

pom.xml
<project>
    ...
    <plugin>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-maven</artifactId>
        <version>2.8</version>
    </plugin>
    ...
</project>

13.6.2. Configuration

Properties handling

Since 2.5 report generation is configurable. There are a few ways to do it:

Directly from source code

Put allure.properties (report.properties for Allure 1.5 or above) to classpath. Both compile class path and test class path are supported.

Or specify the properties in configuration
pom.xml
1
2
3
4
5
6
7
8
9
<plugin>
    <groupId>io.qameta.allure</groupId>
    <artifactId>allure-maven</artifactId>
    <configuration>
       <properties>
           <allure.issues.tracker.pattern>http://example.com/%s</allure.issues.tracker.pattern>
       </properties>
    </configuration>
</plugin>
Or specify the propertiesFilePath
pom.xml
1
2
3
<configuration>
   <propertiesFilePath>path/to/your/allure.properties</propertiesFilePath>
</configuration>
Report version

Since 2.6, the plugin constructs a report using Allure which is downloaded from the default url. Allure is extracted and placed into the .allure folder created in your project directory.

The default Allure version is 2.0.1

pom.xml
1
2
3
<configuration>
    <reportVersion>2.0.1</reportVersion>
</configuration>

You can specify your own url for the download, or you can specify a file path using the system property allure.download.url.

pom.xml
1
2
3
<configuration>
    <allureDownloadUrl>https://example.com/allure/allure-2.0.1.zip</allureDownloadUrl>
</configuration>
Results Directory

The path to Allure results directory. In general it is the directory created by allure adaptor and contains allure files and attachments. This path can be relative from the build directory (for maven it is the target directory) or absolute (absolute only for report mojo).

Default value is "allure-results".

System property allure.results.directory.

pom.xml
1
2
3
<configuration>
    <reportDirectory>allure-report</reportDirectory>
</configuration>

13.6.3. Usage

You can generate a report using one of the following command:

  • mvn allure:serve

Report will be generated into temp folder. Web server with results will start.

  • mvn allure:report

Report will be generated tо directory: target/site/allure-maven/index.html

14. Allure Plugins System

Allure is designed to be highly customizable, because practice shows that many teams across the world may have different metrics, or may have developed unique approaches for testing, their products might demand something completely different to the initial requirements Allure was built to comply. To deal with this kinds of challenges Allure has a plugin system, that gives a lot of flexibility to the report representation.

To be precise, all of the basic functionality is modularized into plugins, and it would be too lengthy to list them all here. But nearly every feature described in the features section [link to allure features section] is implemented internally as a plugin.

14.1. Java Plugin API overview

Let’s take a look into the class diagram of the top classes in the Allure plugins system.

Top interfaces hierarchy

Plugin classes can extend 3 basic interfaces that provide different aspects of functionality:

  • Reader allows to implement a readResults method, which defines the logic of reading results with ResultsVisitor instance from the directory with test results.

  • Aggregator allows to implement aggregate method, which defines the logic of aggregating processed results across all the results folders, and write resulting data to the report directory.

  • Widget this interface allows to implement getData method which again defines the logic of processed results aggregation but this time resulting data is saved into the widget.json file to be used by the widget named accordingly to the value supplied in getName.

  • Context<T> interface with T getValue() method enables to create some utility class to be used for all plugins via a method Configuration.requireContext(Class<T>) available from a Configuration instance, provided as a parameter in all of the 3 methods above.

14.2. JS Plugin API overview

Front-end side of the Allure is built using BackboneJS framework. So some basic understanding of it’s internal mechanisms may sometimes be necessary.

Api is accessible from the allure.api global object. Let’s take a look at the list of functions it provides:

  • addTab(tabName, {title, icon, route, onEnter = notFound} = {}) - can be used to define a new tab for the report which will appear on the left pane menu with name tabName, it’s icon will be defined by a css styles provided in the icon string, route will define an address for a new tab page. onEnter should be a function, that instantiates a View class managing your new tab’s representation.

  • addTranslation(lang, json) - gives you an ability to support multiple languages for naming in tabs, widgets or test case blocks you’ve created. lang is a language key, and json is a json object that contains mappings for string values in the specified language.

  • translate(name, options) - is needed if you generate html code in your plugin and don’t use existing components, provided in allure.components. In the tab example above you have to wrap strings in the template in this function call to enable string translation to be picked up from the global registry. See more in the docs for i18next.

  • addWidget(name, Widget) - is a way to create a new widget on the Overview page of the report. name will define it’s displayed name, and Widget is a View to be added to the widgets grid. Api provides you with a base class for a widget at allure.components.WidgetStatusView, which we will examine later in the Behaviors plugin section. But you may design a widget for your own needs extending from Backbone.Marionette.View, just keep in mind that this widget definition is designed to pop up the data to fill the Model for this View from the widgets.json file, by the key you supplied in name parameter.

  • addTestcaseBlock(view, {position}) - allows to add a View class to the Test Case page, in the one of 3 possible block groups, determined by a position argument. Position can be one of this values: tag, after or before. To understand what kind of information you may attach to the test case page, jump to the section with related features [jump to the features list]

14.3. Writing a skeleton code for a new plugin.

Here we will cover the steps that it usually takes to build a new plugin.

14.3.1. Step 1: Create a new plugin project

Plugin structure

Basically, any plugin will be constituted of two main parts:

  • Java classes that process the report data and produce some results in the report folder.

  • JS script that takes stored results and creates a representation for them on the report’s front-end side, e.g. a widget or an additional tab.

Typical structure of a plugin module would look like this:

/my-plugin
    /src
        /dist
            /static
            allure-plugin.yml
        /main
            /java
                /my.company.plugin
    build.gradle

Here in src\dist\static all the static .js and .css files are stored, and everything under src\main\java is a data processing Java code. allure-plugin.yml - is a configuration file.

Contents of allure-plugin.yml file

This file contains directives in human-readable format that plugin loader will further use to locate resources and connect the plugin.

allure-plugin.yml
id: my-plugin
name: Plugin name goes here
description: More detailed explanation of what does this plugin do.
extensions:
- my.company.allure.CustomPlugin // - Fully qualified names of classes that implement `Extension` interface and comprise data processing functionality.
- my.company.allure.CustomWidget
jsFiles:
- index.js
cssFiles:
- styles.css

Adding allure-plugin-api dependency

To be able to use the API you should simply download the allure-plugin-api dependency from the jcenter repository. To do so add to your project build script:

in Gradle:

dependencies {
    compileOnly('io.qameta.allure:allure-plugin-api:${allureVersion}')
}

in Maven:

        <dependency>
            <groupId>io.qameta.allure</groupId>
            <artifactId>allure-plugin-api</artifactId>
            <version>${allureVersion}</version>
            <scope>provided</scope>
        </dependency>

14.3.2. Step 2: Writing a Java class that processes test results

Let’s consider we have some very simple set of parameterized tests, where typical result will contain captured arguments of test case in the parameters section.

{
  "uuid":"0edd28b1-3c7f-4593-8dda-db9aa004891f",
  "fullName":"io.qameta.allure.animals.AnimalsTest.angryCat",
  "name":"angryCat",
  "status":"passed",
  "stage":"finished",
  "start":1495467840415,
  "stop":1495467840416,
  "parameters":[
    {
      "name":"arg0",
      "value":"Hiss!"
    }
  ]
}

We are preparing to write a fully-fledged new plugin that adds a new tab with some test results representation and creates a widget to place on Overview tab with some digested data. For example, let’s consider a plugin that extracts passed and failed parameters from this parameterized tests, creates a new tab, and a widget where only recent failures are displayed.

We should start with writing a Java class that implements Aggregator and Widget interfaces.

MyPlugin.java
public class MyPlugin implements Aggregator, Widget {

    @Override
    public void aggregate(final Configuration configuration,
                          final List<LaunchResults> launches,
                          final Path outputDirectory) throws IOException {
    final JacksonContext jacksonContext = configuration
        .requireContext(JacksonContext.class);
    final Path dataFolder = Files.createDirectories(outputDirectory.resolve("data"));
    final Path dataFile = dataFolder.resolve("myplugindata.json");
    final Stream<TestResult> resultsStream = launches.stream()
        .flatMap(launch -> launch.getAllResults().stream());
    try (OutputStream os = Files.newOutputStream(dataFile)) {
        jacksonContext.getValue().writeValue(os, extractData(resultsStream));
    }
    }

    private Collection<Map> extractData(final Stream<TestResult> testResults) {
        //extraction logic
    }

    @Override
    public Object getData(Configuration configuration, List<LaunchResults> launches) {
        Stream<TestResult> filteredResults = launches.stream().flatMap(launch -> launch.getAllResults().stream())
                .filter(result -> result.getStatus().equals(Status.FAILED));
        return extractData(filteredResults);
    }

    @Override
    public String getName() {
        return "mywidget";
    }
}

What is happening in the code above?

  1. In aggregate method, data that is extracted from test results in the extractData method is written to the myplugindata.json file that is stored in the report’s data folder. To create a proper .json file a JacksonContext is used to obtain a mapper instance. This data will be displayed on the new tab.

  2. getData method implementation creates data to be used in the new widget, and getName method defines name of the entry for the widgets.json file where this data will be stored.

myplugindata.json
[ {
  "sounds" : [ "Growl!", "Hiss!" ],
  "name" : "angryCat"
}, {
  "sounds" : [ "Oink!", "Meow!" ],
  "name" : "hungryCat"
}, {
  "sounds" : [ "Bark!", "Woof!", "Moo!" ],
  "name" : "bigDog"
} ]
widgets.json
...
"mywidget" : [ {
    "sounds" : [ "Oink!" ],
    "name" : "hungryCat"
  }, {
    "sounds" : [ "Moo!" ],
    "name" : "bigDog"
  } ],
...

14.3.3. Adding an utility Context class

Your plugins may require to share some common utilities that would be wise to make available in on-demand manner. A quick example of such an utility class would be JacksonContext, which can be used to obtain a mapper to serialize Java objects with data into the report JSON files.

public class JacksonContext implements Context<ObjectMapper> {

    private final ObjectMapper mapper;

    public JacksonContext() {
        this.mapper = new ObjectMapper()
                .configure(MapperFeature.USE_WRAPPER_NAME_AS_PROPERTY_NAME, true)
                .setAnnotationIntrospector(new JaxbAnnotationIntrospector(TypeFactory.defaultInstance()))
                .enable(SerializationFeature.INDENT_OUTPUT)
                .disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES)
                .setSerializationInclusion(JsonInclude.Include.NON_NULL);
    }

    @Override
    public ObjectMapper getValue() {
        return mapper;
    }
}

Then, from a plugin class it can be accessed from Configuration instance as in the Step 2.

14.3.4. Step 3: Adding a new tab for the report

Here we switch to the front-end side of the Allure report and start with adding some JavaScript code to the index.js file.

Backbone manages data with Models or Collections, on the previous step we saved the data for our page as a Collection<Map>, so the model for the tab should extend Backbone.Collection. This object will contain the data from the file, specified in the url. Then, for your new tab you need to extend a View class from the base AppLayout class that already contains report’s left navigational menu. It is provided in the global allure object:

var MyTabModel = Backbone.Collection.extend({
    url: 'data/myplugindata.json'
})

class MyLayout extends allure.components.AppLayout {

    initialize() {
        this.model = new MyTabModel();
    }

    loadData() {
        return this.model.fetch();
    }

    getContentView() {
        return new MyView({items: this.model.models});
    }
}

In MyLayout class you can override a getContentView method to define some other View class that will manage the contents of your tab. Below is some simplistic implementation of the View class, template is some template function that returns html template with added data.

const template = function (data) {
    html = '<h3 class="pane__title">My Tab</h3>';
    for (var item of data.items) {
        html += '<p>' + item.attributes.name + ' says: ' + item.attributes.sounds + '</p>';
    }
    return html;
}

var MyView = Backbone.Marionette.View.extend({
    template: template,

    render: function () {
        this.$el.html(this.template(this.options));
        return this;
    }
})

After all that add addTab function call would look like this:

allure.api.addTab('mytab', {
    title: 'My Tab', icon: 'fa fa-trophy',
    route: 'mytab',
    onEnter: (function () {
        return new MyLayout()
    })
});

Which will finally give you a new tab:

Hello world tab example

14.3.5. Step 4: Adding a new widget on the Overview page

To create a new widget you need to implement a small View class that manages data that you put into widgets.json on the step 2. Note, that if you return the data from getData as a collection, it will subsequently be provided to the widget as an array, that can be obtained as this.model.get('items'). In the code below template function defines the actual html to be displayed in the widget.

index.js
class MyWidget extends Backbone.Marionette.View {

    template(data) {
            return widgetTemplate(data)
    }

    serializeData() {
        return {
            items: this.model.get('items'),
        }
    }
}

allure.api.addWidget('mywidget', MyWidget);

This finally gives us a new widget on the Overview dashboard.

A new widget on the Overview

14.3.6. Step 5: Adding translation for strings

Returning to the tab example, it’s very easy to enable translated strings in it. In templates you need to substitute plain text strings for placeholders and use translate function, and also you need to register translations via addTranslation.

const template = function (data) {
    html = '<h3 class="pane__title">' + allure.api.translate(mytab.name) + '</h3>';
    for (var item of data.items) {
        html += '<p>' + item.attributes.name + ' says: ' + item.attributes.sounds + '</p>';
    }
    return html;
}

allure.api.addTranslation('en', {
    mytab: {
        name: 'My Tab',
    }
},
});

allure.api.addTranslation('ru', {
    mytab: {
        name: 'Моя Вкладка',
    }
},
});

14.3.7. Step 6: Adding new sections for test case page

Internally, many Allure features are implemented using plugin api, let’s see how for example links are added to the test case page.

With addTestcaseBlock method you can define a View that you can assume will have a test case object as a Model available at this.model.

A View class:

LinksView.js
import './styles.css';
import {View} from 'backbone.marionette';
import {className} from '../../decorators';
import template from './LinksView.hbs';

@className('pane__section')
class LinksView extends View {
    template = template;

    serializeData() {
        return {
            links: this.model.get('links')
        };
    }
}

Handlebars is used as a template engine:

LinksView.hbs
{{#if links}}
    <h3 class="pane__section-title">{{t 'testCase.links.name'}}</h3>
    {{#each links}}
        <span class="testcase-link">
        {{#if (eq type "issue")}}
            <span class="fa fa-bug"></span>
        {{/if}}
        {{#if (eq type "tms")}}
            <span class="fa fa-database"></span>
        {{/if}}
        <a class="link" href="{{this.url}}" target="_blank">{{name}}</a>
    </span>
    {{/each}}
{{/if}}
index.js
import LinksView from './LinksView';

allure.api.addTestcaseBlock(LinksView, {position: 'before'});

Which adds a links section to the test case:

New test case block

14.3.8. Step 7: Plugin distribution

When you build a plugin, you should come up with the following structure, which then can be copied into plugins folder of the commandline distribution.

\my-plugin
    allure-plugin.yml
    plugin.jar
    \lib
        dependency.jar
    \static
        styles.css
        index.js
  • plugin.jar - is a jar with your compiled plugin classes

  • \lib - all of your pugin’s dependencies should be placed in here

  • \static - a folder containing all static .js and .css files.

Here is a template of gradle build script for a plugin project that uses Java Library Distribution Plugin to package plugin classes and copy files and dependencies into one .zip archive.

build.gradle
repositories {
    jcenter()
}

apply plugin: 'java-library-distribution'

jar {
    archiveName = 'plugin.jar'
}

dependencies {
    compileOnly('io.qameta.allure:allure-plugin-api:2.0-BETA8')
}

14.3.9. Step 8: Enabling a plugin

Allure commandline distribution has a following folder structure:

/allure-commandline
    /bin
    /config
        allure.yml
    /lib
    /plugins
        /behaviors-plugin
        /junit-plugin
        /screen-diff-plugin

Here in plugins folder plugins distributions to use at the report generation reside. By default several plugins are already added to the Allure. Their usage is managed by default build profile configuration file`/config/allure.yml`. In this file, under the section plugins plugin folders to use are listed, so it’s contents should look like this:

allure.yml
plugins:
  - behaviors-plugin
  - junit-plugin
  - screen-diff-plugin

To enable your own plugin, copy folder with distribution to the plugins folder and then add the folder’s name to the corresponding build profile configuration:

/allure-commandline
    /bin
    /config
        allure.yml
    /lib
    /plugins
        /behaviors-plugin
        /junit-plugin
        /screen-diff-plugin
        /my-plugin
allure.yml
plugins:
  - behaviors-plugin
  - junit-plugin
  - screen-diff-plugin
  - my-plugin

14.4. Existing plugins

Several important Allure features are implemented as decoupled plugins that are stored independently under the plugins folder of Allure Commandline distribution. Their usage can be managed by build profiles functionality (jump to the (link to the section)[Commandline configuration]).

Let’s start with studuying how one of the simplest plugins works. In this section we will quickly examine a plugin available in the Allure distribution which allows you to change a logo picture shown in the upper-left corner of the report.

Custom logo in the report

plugin sources directory structure:

/src
    /dist
        allure-plugin.yml
        /static
            custom-logo.svg
            styles.css
    build.gradle
allure-plugin.yml
id: custom-logo
name: Custom logo aggregator
description: The aggregator replaces default Allure logo with a custom one
cssFiles:
  - styles.css
  • custom-logo.svg - is a vector graphics file with a logo to use

  • styles.css - a css file that adds the style, that will override default logotype.

styles.css
.side-nav__brand {
  background: url('custom-logo.svg') no-repeat left center;
  margin-left: 10px;
}

14.4.2. Behaviors

Behaviors plugin is created to support behavior-driven approach in testing with Allure report. Test cases should have Feature and Story labels, that plugin will aggregate and create a widget showing statistics of stories results per every feature, and a new tab, where all test results are grouped by their features and stories.

  • In Java-based adaptors you can mark your tests with @Feature and @Story annotations.

  • In JS-based adaptors you can use allure.feature(featureName) and allure.story(storyName) methods

  • In CucumberJVM Features and Stories are extracted exactly as the tests are organized in Features and Scenarios.

Once your tests are properly labeled, you can start using Behaviors plugin right away since it is included in Allure distribution by default. You can find plugin sources in the plugins folder of the Allure 2 project

Behaviors plugin structure:

/src
    /dist
        allure-plugin.yml
        /static
            index.js
    /main
        /java
            /io.qameta.allure.behaviors
                BehaviorsPlugin.java
    build.gradle

BehavioursPlugin.java

This plugin creates a different representation for the test results tree, that is why it’s BehaviorsPlugin class inherits from a base class providing tree aggregation - AbstractTreeAggregator and implements a Widget interface to prepare the data for a widget on the Overview report dashboard.

  • To provide aggregation classifier, plugin should implement a method List<TreeGroup> getGroups(final TestResult result)

@Override
    protected List<TreeGroup> getGroups(final TestResult result) {
        return Arrays.asList(
                TreeGroup.allByLabel(result, LabelName.FEATURE, DEFAULT_FEATURE),
                TreeGroup.allByLabel(result, LabelName.STORY, DEFAULT_STORY)
        );
    }

This groups will determine a placement for every TestResult when building a tree of regrouped results to store in the behaviors.json file.

  • Another part of BehaviorsPlugin class is a getData method that comprises implementation of Widget interface. This method prepares aggregated information on amount of passed stories per every feature, that will be put into the widget.json file.

index.js

After that, in the index.js api calls for creation of a new tab and creation of a new widget are as follows:

allure.api.addTab('behaviors', {
    title: 'tab.behaviors.name', icon: 'fa fa-list',
    route: 'behaviors(/:testcaseId)',
    onEnter: (function () {
        var routeParams = Array.prototype.slice.call(arguments);
        return new allure.components.TreeLayout({
            routeParams: routeParams,
            tabName: 'tab.behaviors.name',
            baseUrl: 'behaviors',
            url: 'data/behaviors.json'
        });
    })
});

allure.api.addWidget('behaviors', allure.components.WidgetStatusView.extend({
    title: 'widget.behaviors.name',
    baseUrl: 'behaviors',
    showLinks: false,
    showAllText: 'widget.behaviors.showAll'
}));

Note a special TreeLayout component that can be used for displaying all types of data, produced by implementations of AbstractTreeAggregator. Also, a route attribute defines a pattern for links to the test case pages.

Behaviors tab

14.4.3. Junit

Junit plugin works without a front-end part, it is created to enable Allure generator to process junit report xml format into the Allure test results, this plugin is enabled by default, so when you use allure generate command for a folder with junit test results, a report will be generated.