TestOps logo
Try TestOps

Cleanup

Each test launch generates a considerable amount of data which includes following artifacts:

  • test scenarios (stored in the database starting from the version 4.9.x but still cleaned using the rules described in this chapter)
  • fixtures (common blocks for tests like SetUp and tearDown)
  • attachments

You may need this information if the test run is not successful, but for the successfully passed tests this data usually consumes a lot of storage space and provides no useful information as well as very old data on the failed and other tests.

Cleanup rules are working for closed launches only. This means, if you have configured cleanup for an artifact after 24 hours, and you have automatic launches closing policy which closes finished launches after 72 hours, the artifacts in such a launch will be deleted after 72 hours when launch will be closed.

Cleanup rules

Allure TestOps allows you configuring automatic cleanup of old files.

There are two types of cleanup rules:

  • Global cleanup rules
  • Per project cleanup rules

Global cleanup rule is being applied to all the projects created in Allure TestOps instance.

Per project cleanup rules superadd the global cleanup rules, they do not replace the global rules.

Example: if you set up global rule for the deletion of attachments for passed tests after 24 hours, and in the same time on a project level you specify the deletion of the same artifacts after 25 hours, then these rules will be applied to this very project together – files will be deleted after 24 hours and then after 25 hours.

Create a global cleanup rule

To create a global cleanup rule, you need administrator privileges.

  1. Go to Administration → Cleanup policies.
  2. Click the + Create button.
  3. Select the Target and Test status of the target test from the drop-down lists.
  4. Specify the Clean delay (in hours) parameter. It defines number of hours between a test run and the data cleanup.
  5. Click Submit. You can also Edit or Delete created rules.

Create a cleanup rule for a specific project

Global cleanup rules configured by the administrator may not be enough for your project. In this case, you can configure a rule for the current project.

Reminder: Per project rules do not replace the Global rules, they are added to the set of rules defined on the Global level of Allure TestOps.

To create a cleanup rule for the project, you must be the project's Owner.

  1. Open your project.
  2. Go to Settings → Cleanup policies. Here you can see the global cleanup rules for your system.
  3. Click the + Create button.
  4. Specify Target, Test status and Clean delay (in hours) in the same way as above.
  5. Click Submit. You can also Edit or Delete created rules.

How the cleanup works

The cleanup has 2 stages:

  1. The collection of files eligible for the deletion.
  2. Slow deletion of the files marked for cleanup.

Collection of the files eligible for the deletion

The creation of the files eligible for the deletion is a high load request for the database, hence this operation is executed at night.

Allure TestOps performs cleanup tasks at 02:00 for Global rules, and per project rules are triggered at 02:30.

The database request searches for the files which creation time matches the cleanup rule and moves such files records to a special database table, so the files become marked for the cleanup.

In some cases, files from the same test result can fall under different cleanup rules and therefore be deleted at different times. For more details, see Setup and Teardown files.

Slow deletion of the files marked for cleanup

After the files are added to the special list for the cleaning, Allure TestOps routine starts cleaning this list:

  1. Each 5 minutes Allure TestOps requests the storage to delete 1000 of files from the cleanup list.
  2. If storage deletes the file or reports a problem, the records are deleted from the table and such files are considered as deleted.

If your tests generate a lot of files, deletion of 1000 files at a time could be not enough, so your cleanup queue in the database will grow and the storage won't have the space cleaned. To fine tune the cleaning, please refer to the Managing the cleanup batch size section of this very article.

Troubleshooting and fine tuning

Managing the cleanup batch size

Allure TestOps collects the list of files based on the cleanup policy and creates blob remove tasks (a record in the database which points to a certain file), then each 5 minutes Allure TestOps runs the task of blobs removal. Files are deleted in batches. The default value of the batch size is 1000. This means that each 5 minutes Allure TestOps will request the blob storage to delete 1000 files.

If your tests create a huge amount of files, these are stored faster than the cleanup routine deletes the files, so you will observe the growth of used space. You can manage (increase) the batch size for files deletion by adding a special parameter to report service configuration.

How to check

Besides the growth of used space, you must see the growth of the queue of files marked for the deletion.

To understand if the default batch size for files deletion isn't enough do the following:

Run the following SQL against the report service database in the morning (09:00), in the middle of the day (around 14:00), and at the end of the day (say at 19:00).

select count(*) from blob_remove_task;

If by the end of the working day the size of the cleanup queue more than zero, then you need to increase the batch size.

ALLURE_SCHEDULER_CLEANER_BLOB_BATCHSIZE: 2000

If the queue size is several hundreds of thousands or millions of records, then you need to set a very aggressive value for the deletion of the files, but this needs to be done during the minimal workload (weekends or night time), otherwise big batch could affect the overall performance of Allure TestOps, as the during the test results processing, Allure TestOps uses the storage very heavily and test results processing and the deletion operations will compete for the storage resources.

When ALLURE_SCHEDULER_CLEANER_BLOB_BATCHSIZE should be applied

ALLURE_SCHEDULER_CLEANER_BLOB_BATCHSIZE needs to be adjusted only in a case if you see that the size of space used to store the test results artifacts is growing or isn't changing with the time, and there are cleanup rules in place with growing blob remove task queue, i.e. you are creating more files that you are deleting.

The batch size for the files deletion should have reasonable value. Don't increase this parameter violently as it will dramatically degrade the performance of your blob storage and generally performance of Allure TestOps.

Setup and Teardown files

Automated test cases executed using an Allure Report adapter can have setup and teardown methods — i.e., additional steps performed before and after the test itself. Since these methods can have a different result from the main test result (i.e., fail independently), Allure TestOps can apply different cleanup rules to them.

For example, let's assume you have configured two cleanup rules regarding attached files:

  • Delete attachments in successful test results after an hour.
  • Delete attachments in failed test results after 24 hours.

If you upload a test result in which the main part was passed successfully but the teardown step has failed, both rules above will apply. If the main part has attachments, they will be deleted after an hour. If the teardown step has attachments, they will be deleted after 24 hours.

Tips and tricks

Manual triggering of cleanup procedures

At the moment, there is no UI elements in Allure TestOps to trigger cleanup rules, but you can use API commands to initiate the cleanup process outside of the defined schedule.

Open Allure TestOps API description by adding /swagger-ui.html to your Allure TestOps instance URL. Now you need to proceed to the cleanup controller section and select one of the API functions you need to trigger outside the schedule.

To trigger the global rules, you need to execute /cleanup/scheduler/cleaner_schema_global for the global cleanup pules or /cleanup/scheduler/cleaner_schema_project for per project cleanup rules.

Quick creation of cleanup rules for a project

To quickly create the full set of cleanup rules on a project level, you can use this shell script:

ALLURE_TOKEN=<add token here>
ALLURE_ENDPOINT=<add Allure TestOps URL here>
ALLURE_PROJECT_ID=<YOUR PROJECT ID>

TARGET_ARTEFACT="attachment scenario fixture"
TEST_STATUS="passed failed broken unknown skipped"

DELETE_DELAY=48

for ARTEFACT in $TARGET_ARTEFACT
    do
        for STATUS in $TEST_STATUS
        do 
            echo "Marking ${ARTEFACT} for ${STATUS} tests for deletion after ${DELETE_DELAY} after creation \n"
            curl -X POST "${ALLURE_ENDPOINT}/api/rs/cleanerschema" --header "accept: */*" --header "Content-Type: application/json" --header "Authorization: Api-Token ${ALLURE_TOKEN}" -d "{\"projectId\": ${ALLURE_PROJECT_ID},\"status\": \"${STATUS}\",\"target\": \"${ARTEFACT}\",\"delay\": ${DELETE_DELAY}}"
            echo "\n"
        done
    done

You can also use this script for global rules creation, to do this, you need to remove the following from curl data

\"projectId\": ${ALLURE_PROJECT_ID},.

© Qameta Software Inc. All rights reserved.