TestOps logo
Try TestOps

Cleanup

Each test launch generates a considerable amount of data which includes following artifacts:

  • test scenarios;
  • fixtures (common blocks for tests such as SetUp and TearDown);
  • attachments.

Allure TestOps allows you to configure automatic artifact deletion. This can be useful for successful test runs, which often take up a lot of space without providing useful information, as well as for old failed test results.

Cleanup rules

Allure TestOps provides two types of cleanup rules:

  • Global rule – applies to all projects within the Allure TestOps instance.
  • Project-specific rule – applies in addition to a specific project and does not override global rules.

For example, if you set up a global rule for the deletion of attachments for passed tests after 24 hours, and in the same time on a project level you specify the deletion of the same artifacts after 25 hours, then these rules will be applied to this very project together — files will be deleted after 24 hours and then after 25 hours.

Cleanup rules apply only to closed launches. Specifically, if you have an automatic launch closure policy, artifacts will only be deleted after the launch is closed. For example, if artifact cleanup is set to 24 hours, but the launch closes automatically 72 hours after opening, artifacts will be deleted after 72 hours when the launch is closed.

Default global cleanup rules

New Allure TestOps instances have a predefined set of global cleanup rules. This applies to all instances of the cloud version of Allure TestOps and to newly deployed instances of the server version of Allure TestOps without any existing projects.

By default, the following data is deleted:

  • attachments in successful test results — after 1 week (168 hours);
  • attachments in failed test results and results with other statuses — after 1 month (720 hours);
  • fixtures and test result scenarios — after 1 month (720 hours) regardless of the test result status.

Create a global cleanup rule

To create a global cleanup rule, you must have administrator permissions for the Allure TestOps instance.

  1. Go to Administration → Cleanup policies.
  2. Click the + Create button.
  3. Select the Target and Test status of the target test from the drop-down lists.
  4. Specify the Clean delay (in hours) parameter. It defines number of hours between a test run and the data cleanup.
  5. Click Submit. You can also Edit or Delete created rules.

Create a cleanup rule for a specific project

Global cleanup rules configured by the administrator may not be enough for your project. In this case, you can configure an additional rule for the current project.

To create a cleanup rule for a project, you must be the owner of the project.

  1. Open your project.
  2. Go to Settings → Cleanup policies. Here you can see the global cleanup rules for your system.
  3. Click the + Create button.
  4. Specify Target, Test status and Clean delay (in hours) in the same way as above.
  5. Click Submit. You can also Edit or Delete created rules.

How the cleanup works

The cleanup has 2 stages:

  1. The collection of files eligible for the deletion.
  2. Slow deletion of the files marked for cleanup.

Collecting files eligible for deletion

Compiling a list of files eligible for deletion is a high load request for the database, that is why this operation is executed at night.

Allure TestOps runs cleanup tasks at different times for different rule types:

  • Global rules are processed starting at 02:00.
  • Project-specific rules are processed starting at 02:30.

The database request searches for the files which creation time matches the cleanup rule and moves such files records to a special database table, so the files become marked for the cleanup.

In some cases, files from the same test result can fall under different cleanup rules and therefore be deleted at different times. For more details, see Setup and Teardown files.

Slow deletion of the files marked for cleanup

After the cleanup list is compiled, Allure TestOps starts deleting files in batches from this list:

  1. Every 5 minutes Allure TestOps requests the storage to delete a batch of 1 000 files from the cleanup list.
  2. If storage deletes the file or reports a problem, the records are deleted from the table and such files are considered as deleted.

The number of files deleted every 5 minutes depends on your Allure TestOps instance deployment type. The default value of the batch size is 1 000 files.

Troubleshooting and fine tuning

Managing the cleanup batch size

If your tests generate a large number of files, deleting 1 000 files at a time may not be sufficient. As a result, the cleanup queue in the database will constantly grow, and storage will not be freed. To resolve this, you can increase the cleanup batch size.

How to check

To check the cleanup queue size, run the following SQL query against the testops service database in the morning (09:00), in the middle of the day (around 14:00), and at the end of the day (for example, 19:00):

SELECT COUNT(*) FROM blob_remove_task;

If the cleanup queue size is still greater than zero by the end of the workday, you need to increase the batch size of files deleted every 5 minutes. To do this, modify the ALLURE_SCHEDULER_CLEANER_BLOB_BATCHSIZE parameter in the testops service settings, for example: 

ALLURE_SCHEDULER_CLEANER_BLOB_BATCHSIZE: 2000

If the queue size is several hundreds of thousands or millions of records, then you need to set a very aggressive value for the deletion of the files. This needs to be done during the minimal workload (weekends or night time), otherwise big batch could affect the overall performance of Allure TestOps, as the during the test results processing, Allure TestOps uses the storage very heavily and test results processing and the deletion operations will compete for the storage resources.

The batch size for the files deletion should have reasonable value. Don't increase this parameter violently as it will dramatically degrade the performance of your blob storage and generally performance of Allure TestOps.

When to change ALLURE_SCHEDULER_CLEANER_BLOB_BATCHSIZE value

The ALLURE_SCHEDULER_CLEANER_BLOB_BATCHSIZE parameter should be adjusted only if you have configured all necessary cleanup rules, but the storage size still keeps growing or remains unchanged over time. In other words, if you generate more files than you delete.

SetUp and TearDown files

Automated test cases executed using an Allure Report adapter can have SetUp and TearDown methods — i.e., additional steps performed before and after the test itself. Since these methods can have a different result from the main test result (i.e., fail independently), Allure TestOps can apply different cleanup rules to them.

For example, let's assume you have configured two cleanup rules regarding attached files:

  • Delete attachments in successful test results after an hour.
  • Delete attachments in failed test results after 24 hours.

If you upload a test result in which the main part was passed successfully but the teardown step has failed, both rules above will apply. If the main part has attachments, they will be deleted after an hour. If the teardown step has attachments, they will be deleted after 24 hours.

Tips and tricks

Manual triggering of cleanup procedures

There are no elements in the Allure TestOps interface to trigger cleanup rules, but you can use API commands to initiate the cleanup process outside of the defined schedule:

  1. Open Allure TestOps API description by adding /swagger-ui.html to your Allure TestOps instance URL.

  2. Go to the cleanup-controller section and select one of the API functions:

    • /cleanup/scheduler/cleaner_schema_global — to trigger the global cleanup rules;
    • /cleanup/scheduler/cleaner_schema_project — to trigger project-specific cleanup rules.

Quick creation of cleanup rules for a project

To quickly create the full set of cleanup rules on a project level, you can use this shell script:

ALLURE_TOKEN=<your Allure TestOps token>
ALLURE_ENDPOINT=<Allure TestOps URL>
ALLURE_PROJECT_ID=<your project ID>

TARGET_ARTEFACT="attachment scenario fixture"
TEST_STATUS="passed failed broken unknown skipped"

DELETE_DELAY=48

for ARTEFACT in $TARGET_ARTEFACT
    do
        for STATUS in $TEST_STATUS
        do 
            echo "Marking ${ARTEFACT} for ${STATUS} tests for deletion after ${DELETE_DELAY} after creation \n"
            curl -X POST "${ALLURE_ENDPOINT}/api/rs/cleanerschema" --header "accept: */*" --header "Content-Type: application/json" --header "Authorization: Api-Token ${ALLURE_TOKEN}" -d "{\"projectId\": ${ALLURE_PROJECT_ID},\"status\": \"${STATUS}\",\"target\": \"${ARTEFACT}\",\"delay\": ${DELETE_DELAY}}"
            echo "\n"
        done
    done

You can also use this script for global cleanup rules creation. To do this, remove \"projectId\": ${ALLURE_PROJECT_ID}, from the cURL request.

© Qameta Software Inc. All rights reserved.