debian-mirror-gitlab/doc/user/admin_area/monitoring/background_migrations.md

stage	group	info
Data Stores	Database	To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments

Batched background migrations (FREE SELF)

• Introduced in GitLab 13.11.
• Deployed behind a feature flag, disabled by default.
• Enabled by default in GitLab 13.12.
• Enabled on GitLab.com.
• Recommended for production use.
• For GitLab self-managed instances, GitLab administrators can opt to disable it.

There can be risks when disabling released features. Refer to this feature's version history for more details.

To update database tables in batches, GitLab can use batched background migrations. These migrations are created by GitLab developers and run automatically on upgrade. However, such migrations are limited in scope to help with migrating some integer database columns to bigint. This is needed to prevent integer overflow for some tables.

Check the status of background migrations

All migrations must have a Finished status before you upgrade GitLab. You can check the status of existing migrations.

Enable or disable batched background migrations

Batched background migrations are under development but ready for production use. It is deployed behind a feature flag that is enabled by default. GitLab administrators with access to the GitLab Rails console can opt to disable it.

To enable it:

Feature.enable(:execute_batched_migrations_on_schedule)

To disable it:

Feature.disable(:execute_batched_migrations_on_schedule)

Automatic batch size optimization

• Introduced in GitLab 13.12.
• Deployed behind a feature flag, disabled by default.
• Enabled by default in GitLab 13.12.
• Enabled on GitLab.com.
• Recommended for production use.
• For GitLab self-managed instances, GitLab administrators can opt to disable it.

There can be risks when disabling released features. Refer to this feature's version history for more details.

To maximize throughput of batched background migrations (in terms of the number of tuples updated per time unit), batch sizes are automatically adjusted based on how long the previous batches took to complete.

Enable or disable automatic batch size optimization

Automatic batch size optimization for batched background migrations is under development but ready for production use. It is deployed behind a feature flag that is enabled by default. GitLab administrators with access to the GitLab Rails console can opt to disable it.

To enable it:

Feature.enable(:optimize_batched_migrations)

To disable it:

Feature.disable(:optimize_batched_migrations)

Troubleshooting

Database migrations failing because of batched background migration not finished

When updating to GitLab 14.2 or later there might be a database migration failing with a message like:

StandardError: An error has occurred, all later migrations canceled:

Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':
  {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"push_event_payloads", :column_name=>"event_id", :job_arguments=>[["event_id"], ["event_id_convert_to_bigint"]]}

First, check if you have followed the version-specific upgrade instructions for 14.2. If you have, you can manually finish the batched background migration. If you haven't, choose one of the following methods:

1. Rollback and upgrade through one of the required versions before updating to 14.2+.
2. Roll forward, staying on the current version and manually ensuring that the batched migrations complete successfully.

Roll back and follow the required upgrade path

1. Rollback and restore the previously installed version
2. Update to either 14.0.5 or 14.1 before updating to 14.2+
3. Check the status of the batched background migrations and make sure they are all marked as finished before attempting to upgrade again. If any remain marked as active, you can manually finish them.

Roll forward and finish the migrations on the upgraded version

For a deployment with downtime

To run all the batched background migrations, it can take a significant amount of time depending on the size of your GitLab installation.

1. Check the status of the batched background migrations in the database, and manually run them with the appropriate arguments until the status query returns no rows.

2. When the status of all of all them is marked as complete, re-run migrations for your installation.

3. Complete the database migrations from your GitLab upgrade:

   sudo gitlab-rake db:migrate
   

4. Run a reconfigure:

   sudo gitlab-ctl reconfigure
   

5. Finish the upgrade for your installation.

For a no-downtime deployment

As the failing migrations are post-deployment migrations, you can remain on a running instance of the upgraded version and wait for the batched background migrations to finish normally.

1. Check the status of the batched background migration from the error message, and make sure it is listed as finished. If it is still active, either wait until it is done, or manually finish it.
2. Re-run migrations for your installation, so the remaining post-deployment migrations finish.

Manually finishing a batched background migration

Introduced in GitLab 14.1

If you need to manually finish a batched background migration due to an error, you can run:

sudo gitlab-rake gitlab:background_migrations:finalize[<job_class_name>,<table_name>,<column_name>,'<job_arguments>']

Replace the values in angle brackets with the correct arguments. For example, if you receive an error similar to this:

StandardError: An error has occurred, all later migrations canceled:

Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':
  {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"push_event_payloads", :column_name=>"event_id", :job_arguments=>[["event_id"], ["event_id_convert_to_bigint"]]}

Plug the arguments from the error message into the command:

sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,'[["event_id"]\, ["event_id_convert_to_bigint"]]']

If you need to manually run a batched background migration to continue an upgrade, you can check the status in the database and get the arguments from the query results. For example, if the query returns this:

            job_class_name             | table_name | column_name |           job_arguments
---------------------------------------+------------+-------------+------------------------------------
 CopyColumnUsingBackgroundMigrationJob | events     | id          | [["id"], ["id_convert_to_bigint"]]

The results from the query can be plugged into the command:

sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']

The BackfillNamespaceIdForNamespaceRoute batched migration job fails

In GitLab 14.8, the BackfillNamespaceIdForNamespaceRoute batched background migration job may fail to complete. When retried, a 500 Server Error is returned. This issue was resolved in GitLab 14.9.

To resolve this issue, upgrade GitLab from 14.8 to 14.9. You can ignore the failed batch migration until after you update to GitLab 14.9.