Migration from Allure TestOps 4.x to 5.x

The architecture of an Allure TestOps deployment was changed in the 5.x release. Migrating from 4.x to 5.x requires a few manual steps and will involve a downtime for your users.

The purpose of the migration described below is to merge data from the UAA service database into the report service database.

After successful merging:

• the resulting report service database will be used in Allure TestOps version 5.x;
• the UAA service database will no longer be used and can be deleted.

0. Prepare current Allure TestOps instance

Upgrade to the last release of version 4.x.x → 4.26.5

Before proceeding, you need to upgrade your current deployment to 4.26.5. Attempting to upgrade any other release directly to 5.x will brick your Allure TestOps deployment, and our support team will not be able to help you recover your data.

Follow the minor upgrade instructions for 4.x first if you are starting with an older release:

• Kubernetes,
• Docker Compose,
• DEB,
• RPM.

After the upgrade to 4.26.5, please make sure that Allure TestOps works fine without errors.

Make sure your database version is 15 or higher

Allure TestOps version 5.x will not start on PostgreSQL versions below 15.

Before updating Allure TestOps from version 4.26.5 to any release in version 5.x, make sure that your PostgreSQL server is upgraded to version 15 or higher.

1. Prepare new configuration file

With the migration to version 5.x, the architecture of Allure TestOps has changed — we have merged the gateway, UAA, and report services into a single service called testops. As a result, the configuration files to deploy Allure TestOps have been updated.

Download and edit new configuration file

Copy encryption key from old configuration file to new one

All sensitive integration data (credentials used for the integrations) is stored in the database in encrypted form. The sensitive data is encrypted using the AES algorithm, which requires an encryption key. This encryption key must be transferred from the configuration of version 4.x to the configuration of version 5.x. Otherwise, the integrations will stop working and you will need to re-enter all integration secrets.

############################################################################################################
# cryptoPass parameter is used to encrypt sensitive data (passwords, API tokens in the database)
# Come up with some phrase, then use Base64 encoding, and paste encoded value here.
# In this example we user phrase "This phrase must be kept in a dark dry place." and encoded it as Base64
# phrase mustn't be changed over time, otherwise your integrations would stop working and you will need to re-submit data
############################################################################################################
cryptoPass: <use the value of cryptoPass parameter from version 4.x here>

########################################################################################
# TESTOPS_CRYPTO_PASSWORD 
# Parameter is used for the encryption of the sensitive data in the database. Users' passwords,
# Integrations' secrets are encrypted using this password
# Update this parameter once before the deployment and do not change it. Keep it in a safe place.
########################################################################################
TESTOPS_CRYPTO_PASSWORD=<use the value of ALLURE_CRYPTO_PASSWORD parameter from version 4.x here>

########################################################################################
# ALLURE_CRYPTO_PASSWORD 
# Parameter is used for the encryption of the sensitive data in the database. Users' passwords,
# integrations' secrets are encrypted using this value.
# Update this parameter once before the deployment and do not change it. Keep it in a safe place.
########################################################################################
ALLURE_CRYPTO_PASSWORD=<use the value of ALLURE_CRYPTO_PASSWORD parameter from version 4.x here>

2. Shut down 4.26.5 instance

Before proceeding, shut down the Allure TestOps 4.x instance to avoid any data corruption while working with the databases on the next step.

We recommend closing all launches and informing all your users about the upcoming maintenance period.

helm delete allure-testops

docker compose down

sudo systemctl stop allure-gateway allure-uaa allure-report

3. Merge databases

All actions described in this section are mandatory.

Allure TestOps 4.x used two PostgreSQL databases. Allure TestOps 5.x uses a single PostgreSQL database for both user management and test report data.

Before starting Allure TestOps 5.x, you need to migrate the existing data from the UAA service database to the report service database.

The commands below use the pg_dump and pg_restore command-line utilities, along with the following variables:

• password_of_uaa_db_user_here — your password for the UAA service database;
• password_of_report_db_user_here — your password for the report service database;
• --host="db.example.com" — your actual host address.

Back up databases

The way of creating of the database backup depends on your preference and internal procedures. If you are creating regular backups/snapshots of your databases, please create unplanned backup before continuing.

Create full dumps of report and UAA services databases

1. Create a full dump of the report service database:

   # export the password for the database
   export PGPASSWORD='password_of_report_db_user_here'
   # create the database dump
   pg_dump --file=report-dump.sql \
     --host="db.example.com" --port=5432 --username=report --dbname=report
   

2. Create a full dump of the UAA service database:

   # export the password for the database
   export PGPASSWORD='password_of_uaa_db_user_here'
   # create the database dump
   pg_dump --file=uaa-dump.sql \
     --host="db.example.com" --port=5432 --username=uaa --dbname=uaa
   

The resulting files report-dump.sql and uaa-dump.sql will not be used in the merging process. However, we recommend keeping them as a backup to simplify the recovery if something goes wrong.

Create partial dump of UAA service database

This dump will be used in the merging process.

1. Export the password for the UAA service database so pg_dump tool could use it:

   export PGPASSWORD='password_of_uaa_db_user_here'
   

2. Create a partial dump of the UAA service database:

   @hint[warning] In the command below, you need to update database schema in switch --schema if you are using database schema different from public. @end_hint

   Please also make sure that you modified the arguments related to authentication. Removing or modifying other arguments may lead to a failed migration.

   pg_dump --file=uaa_for_merge.dump \
     --host="db.example.com" --username=uaa --role=uaa --dbname="uaa" \
     --schema="public" \
     --exclude-table-data="databasechangelog" \
     --exclude-table-data="databasechangeloglock" \
     --exclude-table-data="shedlock" \
     --format=c --blobs --encoding=UTF8 --compress=9 \
     --no-owner --no-privileges --no-comments  --verbose
   

   Note that the command will produce an extensive log output. Make sure that there are no error messages in the output.

Import partial dump of UAA service database to the report service database

In this step, we are going to load the partial dump created in the previous step into the report service database, as follows:

1. Export the password for the report service database so pg_restore tool could use it:

   export PGPASSWORD='password_of_report_db_user_here'
   

2. Restore the UAA service database partial dump created in the previous step to report service database:

   @hint[warning] In the example below, you need to update database schema in switch --schema if you are using database schema different from public. @end_hint

   pg_restore --host="db.example.com" \
       --username="report" --role="report" --dbname="report" \
       --no-owner --no-privileges --verbose \
       --schema "public" \
       "uaa_for_merge.dump"
   

   Note that the command will produce an extensive log output. Make sure that there are no error messages in the output, except for minor ones:

   • function "now_epoch_mills" already exists;
   • relation "databasechangelog" already exists;
   • relation "databasechangeloglock" already exists;
   • multiple primary keys for table "databasechangeloglock" are not allowed;
   • errors related to the availability of shedlock table.

   If any other error occur on this stage, please consult support team before proceeding any further (select the “Urgent” priority when creating a ticket).

Insert UAA database service changelog to resulting report service database

This step cannot be omitted.

Download the file insert_to_changelog.sql (the changelog for the UAA service database) and run it for the resulting report service database as follows:

1. Export the password for the report service database so psql tool could use it:

   export PGPASSWORD='password_of_report_db_user_here'
   

2. Import the UAA service database changelog into the resulting report service database:

   psql --file="insert_to_changelog.sql" \
     --host="db.example.com" --port=5432 --username=report --dbname=report
   

4. Start 5.x instance

Configure database connection

The database for Allure TestOps version 5.x will be the resulting report service database we got in the previous steps.

docker compose up -d

sudo systemctl start testops