debian-mirror-gitlab/doc/administration/raketasks/storage.md

215 lines
5.6 KiB
Markdown
Raw Normal View History

2018-03-17 18:26:18 +05:30
# Repository Storage Rake Tasks
2020-04-22 19:07:51 +05:30
This is a collection of Rake tasks you can use to help you list and migrate
2019-09-30 21:07:59 +05:30
existing projects and attachments associated with it from Legacy storage to
2018-03-17 18:26:18 +05:30
the new Hashed storage type.
You can read more about the storage types [here][storage-types].
## Migrate existing projects to Hashed storage
2019-09-30 21:07:59 +05:30
Before migrating your existing projects, you should
2018-03-17 18:26:18 +05:30
[enable hashed storage][storage-migration] for the new projects as well.
2019-09-30 21:07:59 +05:30
This task will schedule all your existing projects and attachments associated with it to be migrated to the
2018-03-17 18:26:18 +05:30
**Hashed** storage type:
**Omnibus Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-11-08 19:23:39 +05:30
sudo gitlab-rake gitlab:storage:migrate_to_hashed
2018-03-17 18:26:18 +05:30
```
**Source Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-11-08 19:23:39 +05:30
sudo -u git -H bundle exec rake gitlab:storage:migrate_to_hashed RAILS_ENV=production
```
They both also accept a range as environment variable:
2018-03-17 18:26:18 +05:30
2020-03-13 15:44:24 +05:30
```shell
2018-11-08 19:23:39 +05:30
# to migrate any non migrated project from ID 20 to 50.
2019-09-30 21:07:59 +05:30
export ID_FROM=20
2018-11-08 19:23:39 +05:30
export ID_TO=50
2018-03-17 18:26:18 +05:30
```
2019-07-07 11:18:12 +05:30
You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page.
2019-09-30 21:07:59 +05:30
There is a specific Queue you can watch to see how long it will take to finish:
2019-07-07 11:18:12 +05:30
`hashed_storage:hashed_storage_project_migrate`
2018-03-17 18:26:18 +05:30
2019-09-30 21:07:59 +05:30
After it reaches zero, you can confirm every project has been migrated by running the commands bellow.
2018-03-17 18:26:18 +05:30
If you find it necessary, you can run this migration script again to schedule missing projects.
2019-07-07 11:18:12 +05:30
Any error or warning will be logged in Sidekiq's log file.
2018-03-17 18:26:18 +05:30
2019-07-31 22:56:46 +05:30
NOTE: **Note:**
If Geo is enabled, each project that is successfully migrated generates an event to replicate the changes on any **secondary** nodes.
2020-04-22 19:07:51 +05:30
You only need the `gitlab:storage:migrate_to_hashed` Rake task to migrate your repositories, but we have additional
2018-03-17 18:26:18 +05:30
commands below that helps you inspect projects and attachments in both legacy and hashed storage.
2019-07-07 11:18:12 +05:30
## Rollback from Hashed storage to Legacy storage
If you need to rollback the storage migration for any reason, you can follow the steps described here.
NOTE: **Note:** Hashed Storage will be required in future version of GitLab.
2019-09-30 21:07:59 +05:30
To prevent new projects from being created in the Hashed storage,
2019-07-07 11:18:12 +05:30
you need to undo the [enable hashed storage][storage-migration] changes.
This task will schedule all your existing projects and associated attachments to be rolled back to the
Legacy storage type.
For Omnibus installations, run the following:
2020-03-13 15:44:24 +05:30
```shell
2019-07-07 11:18:12 +05:30
sudo gitlab-rake gitlab:storage:rollback_to_legacy
```
For source installations, run the following:
2020-03-13 15:44:24 +05:30
```shell
2019-07-07 11:18:12 +05:30
sudo -u git -H bundle exec rake gitlab:storage:rollback_to_legacy RAILS_ENV=production
```
Both commands accept a range as environment variable:
2020-03-13 15:44:24 +05:30
```shell
2019-07-07 11:18:12 +05:30
# to rollback any migrated project from ID 20 to 50.
2019-09-30 21:07:59 +05:30
export ID_FROM=20
2019-07-07 11:18:12 +05:30
export ID_TO=50
```
You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page.
On the **Queues** tab, you can watch the `hashed_storage:hashed_storage_project_rollback` queue to see how long the process will take to finish.
2019-09-30 21:07:59 +05:30
After it reaches zero, you can confirm every project has been rolled back by running the commands bellow.
2019-07-07 11:18:12 +05:30
If some projects weren't rolled back, you can run this rollback script again to schedule further rollbacks.
Any error or warning will be logged in Sidekiq's log file.
2018-03-17 18:26:18 +05:30
## List projects on Legacy storage
To have a simple summary of projects using **Legacy** storage:
**Omnibus Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-11-08 19:23:39 +05:30
sudo gitlab-rake gitlab:storage:legacy_projects
2018-03-17 18:26:18 +05:30
```
**Source Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-11-08 19:23:39 +05:30
sudo -u git -H bundle exec rake gitlab:storage:legacy_projects RAILS_ENV=production
2018-03-17 18:26:18 +05:30
```
To list projects using **Legacy** storage:
**Omnibus Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-11-08 19:23:39 +05:30
sudo gitlab-rake gitlab:storage:list_legacy_projects
2018-03-17 18:26:18 +05:30
```
**Source Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-11-08 19:23:39 +05:30
sudo -u git -H bundle exec rake gitlab:storage:list_legacy_projects RAILS_ENV=production
2018-03-17 18:26:18 +05:30
```
## List projects on Hashed storage
To have a simple summary of projects using **Hashed** storage:
**Omnibus Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-11-08 19:23:39 +05:30
sudo gitlab-rake gitlab:storage:hashed_projects
2018-03-17 18:26:18 +05:30
```
**Source Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-11-08 19:23:39 +05:30
sudo -u git -H bundle exec rake gitlab:storage:hashed_projects RAILS_ENV=production
2018-03-17 18:26:18 +05:30
```
To list projects using **Hashed** storage:
**Omnibus Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-11-08 19:23:39 +05:30
sudo gitlab-rake gitlab:storage:list_hashed_projects
2018-03-17 18:26:18 +05:30
```
**Source Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-11-08 19:23:39 +05:30
sudo -u git -H bundle exec rake gitlab:storage:list_hashed_projects RAILS_ENV=production
2018-03-17 18:26:18 +05:30
```
## List attachments on Legacy storage
To have a simple summary of project attachments using **Legacy** storage:
**Omnibus Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-11-08 19:23:39 +05:30
sudo gitlab-rake gitlab:storage:legacy_attachments
2018-03-17 18:26:18 +05:30
```
**Source Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-11-08 19:23:39 +05:30
sudo -u git -H bundle exec rake gitlab:storage:legacy_attachments RAILS_ENV=production
2018-03-17 18:26:18 +05:30
```
To list project attachments using **Legacy** storage:
**Omnibus Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-11-08 19:23:39 +05:30
sudo gitlab-rake gitlab:storage:list_legacy_attachments
2018-03-17 18:26:18 +05:30
```
**Source Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-11-08 19:23:39 +05:30
sudo -u git -H bundle exec rake gitlab:storage:list_legacy_attachments RAILS_ENV=production
2018-03-17 18:26:18 +05:30
```
## List attachments on Hashed storage
To have a simple summary of project attachments using **Hashed** storage:
**Omnibus Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-11-08 19:23:39 +05:30
sudo gitlab-rake gitlab:storage:hashed_attachments
2018-03-17 18:26:18 +05:30
```
**Source Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-11-08 19:23:39 +05:30
sudo -u git -H bundle exec rake gitlab:storage:hashed_attachments RAILS_ENV=production
2018-03-17 18:26:18 +05:30
```
To list project attachments using **Hashed** storage:
**Omnibus Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-11-08 19:23:39 +05:30
sudo gitlab-rake gitlab:storage:list_hashed_attachments
2018-03-17 18:26:18 +05:30
```
**Source Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-11-08 19:23:39 +05:30
sudo -u git -H bundle exec rake gitlab:storage:list_hashed_attachments RAILS_ENV=production
2018-03-17 18:26:18 +05:30
```
[storage-types]: ../repository_storage_types.md
[storage-migration]: ../repository_storage_types.md#how-to-migrate-to-hashed-storage