Test plans

Test plan is a reusable execution scope: a set of test cases that are meant to be executed together for a specific purpose such as smoke testing, release verification, or targeted validation of one product area.

Test plans can be created from manually selected test cases, from filters, or from AQL queries. They are especially useful when the same scope needs to be run repeatedly or needs to be shared between manual and automated execution paths.

How test plans work

A test plan brings execution scope and ownership together: what should run, how the scope is selected and presented, and who or what owns each part.

an optional tree used to preview and organize the scope;
an optional AQL query that defines the intended scope;
an explicit set of selected test cases that are currently included in the plan;
executor assignments that say which user or job should run which part of the plan.

This gives one place to answer both questions:

What belongs in this run?
Who or what should execute it?

Creating test plans

Creating test plan by selecting test cases

To create a test plan by manually selecting test cases:

Go to Test cases.
Select the test cases using Ctrl or by selecting the checkboxes next to them.
Click ⋯ in the bottom panel and select Test plan.
Enter a name for the test plan.
Click Submit. A new test plan will be created in the Test plans section.

Creating test plan using filters

To create a test plan using filters:

Go to Test cases.
Click the search box above the test case list.
Enter the search terms to filter the test cases.
Select the filtered test cases using Ctrl or by selecting the checkboxes next to them.
Click ⋯ in the bottom panel and select Test plan.
Enter a name for the test plan.
Click Submit. A new test plan will be created in the Test plans section.

To learn more about using filters, see the Test cases article.

Creating test plan using AQL

To create a test plan using Allure Query Language:

Go to Test plans.
Click + Test plan.
Under Name, enter a name for the test plan.
Optionally, under Tree, select a tree that will be used to preview the test cases.
Under Test cases query, enter an AQL query to select the test cases for the test plan. The list of test cases matching the query will update automatically.
Select the filtered test cases using Ctrl or by selecting the checkboxes next to them.
Click Submit. A new test plan will be created in the Test plans section.

Use the AQL-based model when the plan should stay defined by metadata rather than by one-time manual selection.

Scope, drift, and explicit inclusion

When a plan is created from AQL, the query defines the intended scope, but the plan still stores an explicit included set of test cases.

This is why a plan can later show drift:

a test case may start matching the query and not yet be included in the plan;
an included test case may stop matching the query after its metadata changes.

When that happens, use Sync test cases to reconcile the current plan contents with the current filter definition.

Managing test plans

All created test plans can be found in the Test plans section of your project.

You can edit and delete test plans by clicking the ⋯ icon next to the test plan. When editing a test plan, you can change its name and the set of test cases of which it consists.

If a test plan was created using an AQL query, you can update its set of test cases (test cases that match the AQL query) by clicking ⋯ and selecting Sync test cases.

Editing test plans

Although it is technically possible, we do not recommend changing the type of a test plan (e.g., switching from manually selected test cases to AQL queries) after you have created it. Changing the creation model later can make the plan scope inconsistent with the rule that originally defined it. So, if you created a test plan based on tag = "smoke" rule, and you want to include one more test case, then you need to update your test case to meet the test plan criteria rather than manually adding the test case to the created test plan.

Go to Test plans.
Find the test plan you want to edit, then click the ⋯ icon next to it and select Edit. You can use the search bar at the top of the page to search for the test plan by its name.
Under Name, enter a new name for the test plan if you want to change it.
If the test plan was created using an AQL query, you can change it under Test cases query. Otherwise, you can ignore this field.
Revise the list of test cases if necessary and click Submit.

Executors: who or what runs the plan

Each test plan has an Executors tab that splits the plan into:

manual execution groups, where test cases are assigned to users;
automated execution groups, where test cases are assigned to jobs.

This is what lets one plan coordinate mixed manual and automated work without forcing users to rebuild the run scope every time.

Assigning team members and jobs

Normally, when you run a single test case, you can assign a team member to the launch if the test case is manual, or specify a job if the test case is automated.

When using test plans, instead of assigning team members and specifying jobs for each test case every time you run a test plan, you can do that beforehand.

Go to Test plans.
Click a test plan to which you want to assign team members or specify jobs.
Go to the Executors tab.
Click a group of test cases under Manual to assign team members or a group under Automated to specify jobs.
On the right side of the page, select the test cases by checking the boxes to the left of their names.
In the upper right of the test case list, click the Bulk actions button and select Assign user (manual test cases) or Assign job (automated test cases).
Select a team member or a job and click Submit.

Executor assignment is especially important for mixed runs and for plans that route different automated subsets to different CI jobs.

Launching test plans

Go to Test plans.
Find the test plan you want to run and click the Run test plan icon next to it. A new window will appear where you can change the attributes of the launch.
Under Name, enter a name for the launch, or keep the suggested name.
Under Tags, you can select tags to be added to the launch.
Under Issues, you can link the launch to one or several issues in external issue trackers.
Under Links, you can add free-form links to the launch.
Under Environment, you can specify one or more sets of environment variables for the launch.
When finished, click the Submit button to run the test plan, creating a new launch in the Launches section.

When the plan contains automated test cases, Allure TestOps uses the assigned jobs and the current plan scope to tell CI which tests should run.

What must be true for selective execution to work

If you are executing a plan that includes automated tests, your framework integration must support selective execution of test cases. Otherwise, when you run a test plan, all of your integrated tests may be executed instead of only the plan scope.

For selective execution to work as intended:

Automated test cases must already exist in Allure TestOps and be linked to the correct job or jobs.
The job must be configured so it can be used to run tests from Allure TestOps.
The framework or adapter must support the test-plan-based execution contract.
If the plan uses several executors, each part of the scope must be assigned to the correct user or job before the run starts.

For the TestOps-to-CI selective execution contract, see allurectl.

When to use a test plan

Use a test plan when:

the same execution scope is reused often;
the scope should be defined by metadata instead of ad hoc selection;
manual and automated execution need to stay in one shared plan;
different automated subsets should be routed to different CI jobs;
the team needs a named, reviewable scope for a release or validation task.

Next steps

Launches for the resulting execution context.

Run automated tests for CI-backed plan execution.

Create combined launches for mixed manual-plus-automation use cases.