2021-01-29 00:20:46 +05:30
---
2022-07-23 23:45:48 +05:30
stage: Systems
2021-02-22 17:27:13 +05:30
group: Distribution
2022-11-25 23:54:43 +05:30
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
2021-01-29 00:20:46 +05:30
---
2021-03-11 19:13:27 +05:30
# Repository storage Rake tasks **(FREE SELF)**
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
This is a collection of Rake tasks to help you list and migrate
existing projects and their attachments to the new
[hashed storage ](../repository_storage_types.md ) that GitLab
uses to organize the Git data.
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
## List projects and attachments
2018-03-17 18:26:18 +05:30
2022-01-26 12:08:38 +05:30
The following Rake tasks lists the projects and attachments that are
2020-05-24 23:13:21 +05:30
available on legacy and hashed storage.
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
### On legacy storage
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
To have a summary and then a list of projects and their attachments using legacy storage:
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
- **Omnibus installation**
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
```shell
# Projects
sudo gitlab-rake gitlab:storage:legacy_projects
sudo gitlab-rake gitlab:storage:list_legacy_projects
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
# Attachments
sudo gitlab-rake gitlab:storage:legacy_attachments
sudo gitlab-rake gitlab:storage:list_legacy_attachments
```
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
- **Source installation**
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
```shell
# Projects
sudo -u git -H bundle exec rake gitlab:storage:legacy_projects RAILS_ENV=production
sudo -u git -H bundle exec rake gitlab:storage:list_legacy_projects RAILS_ENV=production
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
# Attachments
sudo -u git -H bundle exec rake gitlab:storage:legacy_attachments RAILS_ENV=production
sudo -u git -H bundle exec rake gitlab:storage:list_legacy_attachments RAILS_ENV=production
```
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
### On hashed storage
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
To have a summary and then a list of projects and their attachments using hashed storage:
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
- **Omnibus installation**
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
```shell
# Projects
sudo gitlab-rake gitlab:storage:hashed_projects
sudo gitlab-rake gitlab:storage:list_hashed_projects
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
# Attachments
sudo gitlab-rake gitlab:storage:hashed_attachments
sudo gitlab-rake gitlab:storage:list_hashed_attachments
```
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
- **Source installation**
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
```shell
# Projects
sudo -u git -H bundle exec rake gitlab:storage:hashed_projects RAILS_ENV=production
sudo -u git -H bundle exec rake gitlab:storage:list_hashed_projects RAILS_ENV=production
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
# Attachments
sudo -u git -H bundle exec rake gitlab:storage:hashed_attachments RAILS_ENV=production
sudo -u git -H bundle exec rake gitlab:storage:list_hashed_attachments RAILS_ENV=production
```
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
## Migrate to hashed storage
2018-03-17 18:26:18 +05:30
2021-02-22 17:27:13 +05:30
WARNING:
2020-05-24 23:13:21 +05:30
In GitLab 13.0, [hashed storage ](../repository_storage_types.md#hashed-storage )
is enabled by default and the legacy storage is deprecated.
2021-06-08 01:23:25 +05:30
GitLab 14.0 eliminates support for legacy storage. If you're on GitLab
2020-05-24 23:13:21 +05:30
13.0 and later, switching new projects to legacy storage is not possible.
2022-07-16 23:28:13 +05:30
The option to choose between hashed and legacy storage in the Admin Area has
2020-05-24 23:13:21 +05:30
been disabled.
2018-03-17 18:26:18 +05:30
2022-01-26 12:08:38 +05:30
This task must be run on any machine that has Rails/Sidekiq configured, and the task
schedules all your existing projects and attachments associated with it to be
2021-02-22 17:27:13 +05:30
migrated to the **Hashed** storage type:
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
- **Omnibus installation**
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
```shell
sudo gitlab-rake gitlab:storage:migrate_to_hashed
```
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
- **Source installation**
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
```shell
sudo -u git -H bundle exec rake gitlab:storage:migrate_to_hashed RAILS_ENV=production
```
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
If you have any existing integration, you may want to do a small rollout first,
to validate. You can do so by specifying an ID range with the operation by using
the environment variables `ID_FROM` and `ID_TO` . For example, to limit the rollout
to project IDs 50 to 100 in an Omnibus GitLab installation:
2018-03-17 18:26:18 +05:30
2020-03-13 15:44:24 +05:30
```shell
2020-05-24 23:13:21 +05:30
sudo gitlab-rake gitlab:storage:migrate_to_hashed ID_FROM=50 ID_TO=100
2018-03-17 18:26:18 +05:30
```
2021-09-04 01:27:46 +05:30
To monitor the progress in GitLab:
2018-03-17 18:26:18 +05:30
2022-10-11 01:57:18 +05:30
1. On the top bar, select **Main menu > Admin** .
2021-09-04 01:27:46 +05:30
1. On the left sidebar, select **Monitoring > Background Jobs** .
1. Watch how long the `hashed_storage:hashed_storage_project_migrate` queue
2022-01-26 12:08:38 +05:30
takes to finish. After it reaches zero, you can confirm every project
2021-09-04 01:27:46 +05:30
has been migrated by running the commands above.
If you find it necessary, you can run the previous migration script again to schedule missing projects.
2018-03-17 18:26:18 +05:30
2021-06-08 01:23:25 +05:30
Any error or warning is logged in Sidekiq's log file.
2018-03-17 18:26:18 +05:30
2020-11-24 15:15:51 +05:30
If [Geo ](../geo/index.md ) is enabled, each project that is successfully migrated
2020-05-24 23:13:21 +05:30
generates an event to replicate the changes on any **secondary** nodes.
2018-03-17 18:26:18 +05:30
2021-06-08 01:23:25 +05:30
You only need the `gitlab:storage:migrate_to_hashed` Rake task to migrate your repositories, but there are
2021-09-04 01:27:46 +05:30
[additional commands ](#list-projects-and-attachments ) to help you inspect projects and attachments in both legacy and hashed storage.
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
## Rollback from hashed storage to legacy storage
2018-03-17 18:26:18 +05:30
2021-02-22 17:27:13 +05:30
WARNING:
2020-05-24 23:13:21 +05:30
In GitLab 13.0, [hashed storage ](../repository_storage_types.md#hashed-storage )
is enabled by default and the legacy storage is deprecated.
2021-06-08 01:23:25 +05:30
GitLab 14.0 eliminates support for legacy storage. If you're on GitLab
2020-05-24 23:13:21 +05:30
13.0 and later, switching new projects to legacy storage is not possible.
2022-07-16 23:28:13 +05:30
The option to choose between hashed and legacy storage in the Admin Area has
2020-05-24 23:13:21 +05:30
been disabled.
2018-03-17 18:26:18 +05:30
2021-06-08 01:23:25 +05:30
This task schedules all your existing projects and associated attachments to be rolled back to the
2020-05-24 23:13:21 +05:30
legacy storage type.
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
- **Omnibus installation**
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
```shell
sudo gitlab-rake gitlab:storage:rollback_to_legacy
```
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
- **Source installation**
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
```shell
sudo -u git -H bundle exec rake gitlab:storage:rollback_to_legacy RAILS_ENV=production
```
2018-03-17 18:26:18 +05:30
2020-05-24 23:13:21 +05:30
If you have any existing integration, you may want to do a small rollback first,
to validate. You can do so by specifying an ID range with the operation by using
the environment variables `ID_FROM` and `ID_TO` . For example, to limit the rollout
to project IDs 50 to 100 in an Omnibus GitLab installation:
2018-03-17 18:26:18 +05:30
2020-03-13 15:44:24 +05:30
```shell
2020-05-24 23:13:21 +05:30
sudo gitlab-rake gitlab:storage:rollback_to_legacy ID_FROM=50 ID_TO=100
2018-03-17 18:26:18 +05:30
```
2020-10-24 23:57:45 +05:30
You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page.
2022-01-26 12:08:38 +05:30
On the **Queues** tab, you can watch the `hashed_storage:hashed_storage_project_rollback` queue to see how long the process takes to finish.
2018-03-17 18:26:18 +05:30
2020-11-24 15:15:51 +05:30
After it reaches zero, you can confirm every project has been rolled back by running the commands above.
2020-05-24 23:13:21 +05:30
If some projects weren't rolled back, you can run this rollback script again to schedule further rollbacks.
2021-06-08 01:23:25 +05:30
Any error or warning is logged in Sidekiq's log file.
2018-03-17 18:26:18 +05:30
2022-01-26 12:08:38 +05:30
If you have a Geo setup, the rollback is not reflected automatically
2020-05-24 23:13:21 +05:30
on the **secondary** node. You may need to wait for a backfill operation to kick-in and remove
the remaining repositories from the special `@hashed/` folder manually.
2022-07-16 23:28:13 +05:30
## Troubleshooting
The Rake task might not be able to complete the migration to hashed storage.
Checks on the instance will continue to report that there is legacy data:
```plaintext
* Found 1 projects using Legacy Storage
- janedoe/testproject (id: 1234)
```
If you have a subscription, [raise a ticket with GitLab support ](https://support.gitlab.com )
as most of the fixes are relatively high risk, involving running code on the Rails console.
### Read only projects
2022-11-25 23:54:43 +05:30
If you have set projects as read only they might fail to migrate.
2022-07-16 23:28:13 +05:30
1. [Start a Rails console ](../operations/rails_console.md#starting-a-rails-console-session ).
1. Check if the project is read only:
```ruby
project = Project.find_by_full_path('janedoe/testproject')
project.repository_read_only
```
1. If it returns `true` (not `nil` or `false` ), set it writable:
```ruby
project.update!(repository_read_only: false)
```
1. [Re-run the migration Rake task ](#migrate-to-hashed-storage ).
1. Set the project read-only again:
```ruby
project.update!(repository_read_only: true)
```
### Projects pending deletion
Check the project details in the Admin Area. If deleting the project failed
it will show as `Marked For Deletion At ..` , `Scheduled Deletion At ..` and
`pending removal` , but the dates will not be recent.
Delete the project using the Rails console:
1. [Start a Rails console ](../operations/rails_console.md#starting-a-rails-console-session ).
1. With the following code, select the project to be deleted and account to action it:
```ruby
project = Project.find_by_full_path('janedoe/testproject')
user = User.find_by_username('admin_handle')
puts "\nproject selected for deletion is:\nID: #{project.id}\nPATH: #{project.full_path}\nNAME: #{project.name}\n\n"
```
- Replace `janedoe/testproject` with your project path from the Rake take output or from the Admin Area.
- Replace `admin_handle` with the handle of an instance administrator or with `root` .
- Verify the output before proceeding. **There are no other checks performed** .
2022-11-25 23:54:43 +05:30
1. [Destroy the project ](../../user/project/working_with_projects.md#delete-a-project-using-console ) **immediately** :
2022-07-16 23:28:13 +05:30
```ruby
Projects::DestroyService.new(project, user).execute
```
If destroying the project generates a stack trace relating to encryption or the error `OpenSSL::Cipher::CipherError` :
1. [Verify your GitLab secrets ](check.md#verify-database-values-can-be-decrypted-using-the-current-secrets ).
1. If the affected projects have secrets that cannot be decrypted it will be necessary to remove those specific secrets.
[Our documentation for dealing with lost secrets ](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost )
is for loss of all secrets, but it's possible for specific projects to be affected. For example,
to [reset specific runner registration tokens ](../../raketasks/backup_restore.md#reset-runner-registration-tokens )
for a specific project ID:
```sql
UPDATE projects SET runners_token = null, runners_token_encrypted = null where id = 1234;
```
### `Repository cannot be moved from` errors in Sidekiq log
Projects might fail to migrate with errors in the Sidekiq log:
```shell
# grep 'Repository cannot be moved' /var/log/gitlab/sidekiq/current
{"severity":"ERROR","time":"2021-02-29T02:29:02.021Z","message":"Repository cannot be moved from 'janedoe/testproject' to '@hashed< value > ' (PROJECT_ID=1234)"}
```
This might be caused by [a bug ](https://gitlab.com/gitlab-org/gitlab/-/issues/259605 ) in the original code for hashed storage migration.
[There is a workaround for projects still affected by this issue ](https://gitlab.com/-/snippets/2039252 ).