Clean-up

Each test launch generates a considerable amount of data which includes following artifacts:

test scenarios (stored in the database starting from the version 4.9.x but still cleaned using the rules described in this chapter)
fixtures (common blocks for tests like SetUp and tearDown)
attachments

You may need this information if the test run is not successful, but for the successfully passed tests this data usually consumes a lot of storage space and provides no useful information as well as very old data on the failed and other tests.

Cleanup rules are working for closed launches only. This means, if you have configured cleanup for an artifact after 24 hours, and you have automatic launches closing policy which closes finished launches after 72 hours, the artifacts in such a launch will be deleted after 72 hours when launch will be closed.

Clean-up rules

Allure TestOps allows you configuring automatic cleanup of old files.

There are two types of cleanup rules:

Global cleanup rules
Per project cleanup rules

Global cleanup rule is being applied to all the projects created in Allure TestOps instance.

Per project cleanup rules superadd the global cleanup rules, they do not replace the global rules.

Example: if you set up global rule for the deletion of attachments for passed tests after 24 hours, and in the same time on a project level you specify the deletion of the same artifacts after 25 hours, then these rules will be applied to this very project together – files will be deleted after 24 hours and then after 25 hours.

Create a global cleanup rule

To create a global cleanup rule, you need administrator privileges.

In the Allure TestOps main page, go to the User Menu and select Administration in the drop-down menu.
Go to the Clean up policies section.
Click the Create button. Select the Target and Test status of the target test from the drop-down lists.
Specify the Clean delay (in hours) parameter. It defines number of hours between a test run and the data cleanup.
Click Submit. You can also Edit or Delete created rules.

Create a cleanup rule for a specific project

Global cleanup rules configured by the administrator may not be enough for your project. In this case, you can configure a rule for the current project.

Reminder: Per project rules do not replace the Global rules, they are added to the set of rules defined on the Global level of Allure TestOps.

To create a cleanup rule for the project, you must be the project's Owner.

Open the required project and go to the Settings section.
Go to the Clean up policies section. Here you can see the global cleanup rules for your system.
Click the Create button and specify Target, Test status and Clean delay (in hours) in the same way as above.
Click Submit. You can also Edit or Delete created rules.

How the clean-up works

The clean-up has 2 stages:

The collection of files eligible for the deletion.
Slow deletion of the files marked for clean-up.

Collection of the files eligible for the deletion

The creation of the files eligible for the deletion is a high load request for the database, hence this operation is executed at night.

Allure TestOps performs clean-up tasks at 02:00 for Global rules, and per project rules are triggered at 02:30.

The database request searches for the files which creation time matches the clean-up rule and moves such files records to a special database table, so the files become marked for the clean-up.

Slow deletion of the files marked for clean-up

After the files are added to the special list for the cleaning, Allure TestOps routine starts cleaning this list:

Each 5 minutes Allure TestOps requests the storage to delete 1000 of files from the clean-up list.
If storage deletes the file or reports a problem, the records are deleted from the table and such files are considered as deleted.

If your tests generate a lot of files, deletion of 1000 files at a time could be not enough, so your clean-up queue in the database will grow and the storage won't have the space cleaned. To fine tune the cleaning, please refer to the Managing the clean up batch size section of this very article.

Troubleshooting and fine tuning

Managing the clean up batch size

Allure TestOps collects the list of files based on the clean-up policy and creates blob remove tasks (a record in the database which points to a certain file), then each 5 minutes Allure TestOps runs the task of blobs removal. Files are deleted in batches. The default value of the batch size is 1000. This means that each 5 minutes Allure TestOps will request the blob storage to delete 1000 files.

If your tests create a huge amount of files, these are stored faster than the clean-up routine deletes the files, so you will observe the growth of used space. You can manage (increase) the batch size for files deletion by adding a special parameter to report service configuration.

How to check

Besides the growth of used space, you must see the growth of the queue of files marked for the deletion.

To understand if the default batch size for files deletion isn't enough do the following:

Run the following SQL against the report service database in the morning (09:00), in the middle of the day (around 14:00), and at the end of the day (say at 19:00).

select count(*) from blob_remove_task;

If by the end of the working day the size of the clean-up queue more than zero, then you need to increase the batch size.

ALLURE_SCHEDULER_CLEANER_BLOB_BATCHSIZE: 2000

If the queue size is several hundreds of thousands or millions of records, then you need to set a very aggressive value for the deletion of the files, but this needs to be done during the minimal workload (weekends or night time), otherwise big batch could affect the overall performance of Allure TestOps, as the during the test results processing, Allure TestOps uses the storage very heavily and test results processing and the deletion operations will compete for the storage resources.

When ALLURE_SCHEDULER_CLEANER_BLOB_BATCHSIZE should be applied

ALLURE_SCHEDULER_CLEANER_BLOB_BATCHSIZE needs to be adjusted only in a case if you see that the size of space used to store the test results artifacts is growing or isn't changing with the time, and there are clean up rules in place with growing blob remove task queue, i.e. you are creating more files that you are deleting.

The batch size for the files deletion should have reasonable value. Don't increase this parameter violently as it will dramatically degrade the performance of your blob storage and generally performance of Allure TestOps.

Tips and tricks

Manual triggering of clean up procedures

At the moment, there is no UI elements in Allure TestOps to trigger clean up rules, but you can use API commands to initiate the clean up process outside of the defined schedule.

Open Allure TestOps API description by adding /swagger-ui.html to your Allure TestOps instance URL. Now you need to proceed to the cleanup-controller section and select one of the API functions you need to trigger outside the schedule.

To trigger the global rules, you need to execute /cleanup/scheduler/cleaner_schema_global for the global clean-up pules or /cleanup/scheduler/cleaner_schema_project for per project clean-up rules.

Quick creation of clean-up rules for a project

To quickly create the full set of clean-up rules on a project level, you can use this shell script:

ALLURE_TOKEN=<add token here>
ALLURE_ENDPOINT=<add Allure TestOps URL here>
ALLURE_PROJECT_ID=<YOUR PROJECT ID>

TARGET_ARTEFACT="attachment scenario fixture"
TEST_STATUS="passed failed broken unknown skipped"

DELETE_DELAY=48

for ARTEFACT in $TARGET_ARTEFACT
    do
        for STATUS in $TEST_STATUS
        do 
            echo "Marking ${ARTEFACT} for ${STATUS} tests for deletion after ${DELETE_DELAY} after creation \n"
            curl -X POST "${ALLURE_ENDPOINT}/api/rs/cleanerschema" --header "accept: */*" --header "Content-Type: application/json" --header "Authorization: Api-Token ${ALLURE_TOKEN}" -d "{\"projectId\": ${ALLURE_PROJECT_ID},\"status\": \"${STATUS}\",\"target\": \"${ARTEFACT}\",\"delay\": ${DELETE_DELAY}}"
            echo "\n"
        done
    done

You can also use this script for global rules creation, to do this, you need to remove the following from curl data

\"projectId\": ${ALLURE_PROJECT_ID},.