debian-mirror-gitlab/doc/administration/raketasks/storage.md

169 lines
6.2 KiB
Markdown
Raw Normal View History

2021-01-29 00:20:46 +05:30
---
2021-02-22 17:27:13 +05:30
stage: Enablement
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
2021-01-29 00:20:46 +05:30
---
2021-03-11 19:13:27 +05:30
# Repository storage Rake tasks **(FREE SELF)**
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
This is a collection of Rake tasks to help you list and migrate
existing projects and their attachments to the new
[hashed storage](../repository_storage_types.md) that GitLab
uses to organize the Git data.
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
## List projects and attachments
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
The following Rake tasks will list the projects and attachments that are
available on legacy and hashed storage.
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
### On legacy storage
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
To have a summary and then a list of projects and their attachments using legacy storage:
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
- **Omnibus installation**
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
```shell
# Projects
sudo gitlab-rake gitlab:storage:legacy_projects
sudo gitlab-rake gitlab:storage:list_legacy_projects
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
# Attachments
sudo gitlab-rake gitlab:storage:legacy_attachments
sudo gitlab-rake gitlab:storage:list_legacy_attachments
```
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
- **Source installation**
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
```shell
# Projects
sudo -u git -H bundle exec rake gitlab:storage:legacy_projects RAILS_ENV=production
sudo -u git -H bundle exec rake gitlab:storage:list_legacy_projects RAILS_ENV=production
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
# Attachments
sudo -u git -H bundle exec rake gitlab:storage:legacy_attachments RAILS_ENV=production
sudo -u git -H bundle exec rake gitlab:storage:list_legacy_attachments RAILS_ENV=production
```
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
### On hashed storage
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
To have a summary and then a list of projects and their attachments using hashed storage:
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
- **Omnibus installation**
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
```shell
# Projects
sudo gitlab-rake gitlab:storage:hashed_projects
sudo gitlab-rake gitlab:storage:list_hashed_projects
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
# Attachments
sudo gitlab-rake gitlab:storage:hashed_attachments
sudo gitlab-rake gitlab:storage:list_hashed_attachments
```
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
- **Source installation**
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
```shell
# Projects
sudo -u git -H bundle exec rake gitlab:storage:hashed_projects RAILS_ENV=production
sudo -u git -H bundle exec rake gitlab:storage:list_hashed_projects RAILS_ENV=production
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
# Attachments
sudo -u git -H bundle exec rake gitlab:storage:hashed_attachments RAILS_ENV=production
sudo -u git -H bundle exec rake gitlab:storage:list_hashed_attachments RAILS_ENV=production
```
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
## Migrate to hashed storage
2018-03-17 18:26:18 +05:30
2021-02-22 17:27:13 +05:30
WARNING:
2020-05-24 23:13:21 +05:30
In GitLab 13.0, [hashed storage](../repository_storage_types.md#hashed-storage)
is enabled by default and the legacy storage is deprecated.
2021-06-08 01:23:25 +05:30
GitLab 14.0 eliminates support for legacy storage. If you're on GitLab
2020-05-24 23:13:21 +05:30
13.0 and later, switching new projects to legacy storage is not possible.
The option to choose between hashed and legacy storage in the admin area has
been disabled.
2018-03-17 18:26:18 +05:30
2021-03-08 18:12:59 +05:30
This task must be run on any machine that has Rails/Sidekiq configured and will
schedule all your existing projects and attachments associated with it to be
2021-02-22 17:27:13 +05:30
migrated to the **Hashed** storage type:
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
- **Omnibus installation**
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
```shell
sudo gitlab-rake gitlab:storage:migrate_to_hashed
```
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
- **Source installation**
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
```shell
sudo -u git -H bundle exec rake gitlab:storage:migrate_to_hashed RAILS_ENV=production
```
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
If you have any existing integration, you may want to do a small rollout first,
to validate. You can do so by specifying an ID range with the operation by using
the environment variables `ID_FROM` and `ID_TO`. For example, to limit the rollout
to project IDs 50 to 100 in an Omnibus GitLab installation:
2018-03-17 18:26:18 +05:30
2020-03-13 15:44:24 +05:30
```shell
2020-05-24 23:13:21 +05:30
sudo gitlab-rake gitlab:storage:migrate_to_hashed ID_FROM=50 ID_TO=100
2018-03-17 18:26:18 +05:30
```
2020-10-24 23:57:45 +05:30
You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page.
2020-05-24 23:13:21 +05:30
There is a specific queue you can watch to see how long it will take to finish:
`hashed_storage:hashed_storage_project_migrate`.
2018-03-17 18:26:18 +05:30
2020-11-24 15:15:51 +05:30
After it reaches zero, you can confirm every project has been migrated by running the commands above.
2020-05-24 23:13:21 +05:30
If you find it necessary, you can run this migration script again to schedule missing projects.
2018-03-17 18:26:18 +05:30
2021-06-08 01:23:25 +05:30
Any error or warning is logged in Sidekiq's log file.
2018-03-17 18:26:18 +05:30
2020-11-24 15:15:51 +05:30
If [Geo](../geo/index.md) is enabled, each project that is successfully migrated
2020-05-24 23:13:21 +05:30
generates an event to replicate the changes on any **secondary** nodes.
2018-03-17 18:26:18 +05:30
2021-06-08 01:23:25 +05:30
You only need the `gitlab:storage:migrate_to_hashed` Rake task to migrate your repositories, but there are
[additional commands(#list-projects-and-attachments) to help you inspect projects and attachments in both legacy and hashed storage.
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
## Rollback from hashed storage to legacy storage
2018-03-17 18:26:18 +05:30
2021-02-22 17:27:13 +05:30
WARNING:
2020-05-24 23:13:21 +05:30
In GitLab 13.0, [hashed storage](../repository_storage_types.md#hashed-storage)
is enabled by default and the legacy storage is deprecated.
2021-06-08 01:23:25 +05:30
GitLab 14.0 eliminates support for legacy storage. If you're on GitLab
2020-05-24 23:13:21 +05:30
13.0 and later, switching new projects to legacy storage is not possible.
The option to choose between hashed and legacy storage in the admin area has
been disabled.
2018-03-17 18:26:18 +05:30
2021-06-08 01:23:25 +05:30
This task schedules all your existing projects and associated attachments to be rolled back to the
2020-05-24 23:13:21 +05:30
legacy storage type.
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
- **Omnibus installation**
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
```shell
sudo gitlab-rake gitlab:storage:rollback_to_legacy
```
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
- **Source installation**
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
```shell
sudo -u git -H bundle exec rake gitlab:storage:rollback_to_legacy RAILS_ENV=production
```
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
If you have any existing integration, you may want to do a small rollback first,
to validate. You can do so by specifying an ID range with the operation by using
the environment variables `ID_FROM` and `ID_TO`. For example, to limit the rollout
to project IDs 50 to 100 in an Omnibus GitLab installation:
2018-03-17 18:26:18 +05:30
2020-03-13 15:44:24 +05:30
```shell
2020-05-24 23:13:21 +05:30
sudo gitlab-rake gitlab:storage:rollback_to_legacy ID_FROM=50 ID_TO=100
2018-03-17 18:26:18 +05:30
```
2020-10-24 23:57:45 +05:30
You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page.
2020-05-24 23:13:21 +05:30
On the **Queues** tab, you can watch the `hashed_storage:hashed_storage_project_rollback` queue to see how long the process will take to finish.
2018-03-17 18:26:18 +05:30
2020-11-24 15:15:51 +05:30
After it reaches zero, you can confirm every project has been rolled back by running the commands above.
2020-05-24 23:13:21 +05:30
If some projects weren't rolled back, you can run this rollback script again to schedule further rollbacks.
2021-06-08 01:23:25 +05:30
Any error or warning is logged in Sidekiq's log file.
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
If you have a Geo setup, the rollback will not be reflected automatically
on the **secondary** node. You may need to wait for a backfill operation to kick-in and remove
the remaining repositories from the special `@hashed/` folder manually.