debian-mirror-gitlab/doc/administration/raketasks/uploads/migrate.md

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

198 lines
7.8 KiB
Markdown
Raw Normal View History

2021-01-29 00:20:46 +05:30
---
2021-02-22 17:27:13 +05:30
stage: Enablement
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
2021-01-29 00:20:46 +05:30
---
2021-03-11 19:13:27 +05:30
# Uploads migrate Rake tasks **(FREE SELF)**
2018-05-09 12:01:36 +05:30
2020-11-24 15:15:51 +05:30
There is a Rake task for migrating uploads between different storage types.
- Migrate all uploads with [`gitlab:uploads:migrate:all`](#all-in-one-rake-task) or
- To only migrate specific upload types, use [`gitlab:uploads:migrate`](#individual-rake-tasks).
2018-05-09 12:01:36 +05:30
2020-05-24 23:13:21 +05:30
## Migrate to object storage
2018-05-09 12:01:36 +05:30
2021-02-22 17:27:13 +05:30
After [configuring the object storage](../../uploads.md#using-object-storage) for uploads
to GitLab, use this task to migrate existing uploads from the local storage to the remote storage.
2020-05-24 23:13:21 +05:30
2021-02-22 17:27:13 +05:30
All of the processing is done in a background worker and requires **no downtime**.
2018-12-05 23:21:45 +05:30
2021-01-29 00:20:46 +05:30
Read more about using [object storage with GitLab](../../object_storage.md).
2020-04-22 19:07:51 +05:30
### All-in-one Rake task
2020-05-24 23:13:21 +05:30
GitLab provides a wrapper Rake task that migrates all uploaded files (for example avatars, logos,
attachments, and favicon) to object storage in one step. The wrapper task invokes individual Rake
tasks to migrate files falling under each of these categories one by one.
These [individual Rake tasks](#individual-rake-tasks) are described in the next section.
To migrate all uploads from local storage to object storage, run:
2018-12-05 23:21:45 +05:30
**Omnibus Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-12-05 23:21:45 +05:30
gitlab-rake "gitlab:uploads:migrate:all"
```
**Source Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-12-05 23:21:45 +05:30
sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:migrate:all
```
2022-01-26 12:08:38 +05:30
You can optionally track progress and verify that all packages migrated successfully using the
[PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database):
- `sudo gitlab-rails dbconsole` for Omnibus GitLab instances.
- `sudo -u git -H psql -d gitlabhq_production` for source-installed instances.
Verify `objectstg` below (where `store=2`) has count of all artifacts:
```shell
gitlabhq_production=# SELECT count(*) AS total, sum(case when store = '1' then 1 else 0 end) AS filesystem, sum(case when store = '2' then 1 else 0 end) AS objectstg FROM uploads;
total | filesystem | objectstg
------+------------+-----------
2409 | 0 | 2409
```
Verify that there are no files on disk in the `uploads` folder:
```shell
sudo find /var/opt/gitlab/gitlab-rails/uploads -type f | grep -v tmp | wc -l
```
2020-04-22 19:07:51 +05:30
### Individual Rake tasks
2018-05-09 12:01:36 +05:30
2020-05-24 23:13:21 +05:30
If you already ran the [all-in-one Rake task](#all-in-one-rake-task), there is no need to run these
individual tasks.
The Rake task uses three parameters to find uploads to migrate:
2018-12-05 23:21:45 +05:30
2020-05-24 23:13:21 +05:30
| Parameter | Type | Description |
|:-----------------|:--------------|:-------------------------------------------------------|
| `uploader_class` | string | Type of the uploader to migrate from. |
| `model_class` | string | Type of the model to migrate from. |
| `mount_point` | string/symbol | Name of the model's column the uploader is mounted on. |
2018-05-09 12:01:36 +05:30
2021-02-22 17:27:13 +05:30
NOTE:
These parameters are mainly internal to the structure of GitLab, you may want to refer to the task list
2020-05-24 23:13:21 +05:30
instead below.
2018-05-09 12:01:36 +05:30
2020-05-24 23:13:21 +05:30
This task also accepts an environment variable which you can use to override
the default batch size:
2018-12-05 23:21:45 +05:30
2020-05-24 23:13:21 +05:30
| Variable | Type | Description |
|:---------|:--------|:--------------------------------------------------|
| `BATCH` | integer | Specifies the size of the batch. Defaults to 200. |
2018-05-09 12:01:36 +05:30
2020-05-24 23:13:21 +05:30
The following shows how to run `gitlab:uploads:migrate` for individual types of uploads.
2018-05-09 12:01:36 +05:30
2018-12-05 23:21:45 +05:30
**Omnibus Installation**
2018-05-09 12:01:36 +05:30
2020-03-13 15:44:24 +05:30
```shell
2018-05-09 12:01:36 +05:30
# gitlab-rake gitlab:uploads:migrate[uploader_class, model_class, mount_point]
# Avatars
gitlab-rake "gitlab:uploads:migrate[AvatarUploader, Project, :avatar]"
gitlab-rake "gitlab:uploads:migrate[AvatarUploader, Group, :avatar]"
gitlab-rake "gitlab:uploads:migrate[AvatarUploader, User, :avatar]"
# Attachments
gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Note, :attachment]"
gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :logo]"
gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :header_logo]"
2018-12-05 23:21:45 +05:30
# Favicon
gitlab-rake "gitlab:uploads:migrate[FaviconUploader, Appearance, :favicon]"
2018-05-09 12:01:36 +05:30
# Markdown
gitlab-rake "gitlab:uploads:migrate[FileUploader, Project]"
gitlab-rake "gitlab:uploads:migrate[PersonalFileUploader, Snippet]"
gitlab-rake "gitlab:uploads:migrate[NamespaceFileUploader, Snippet]"
gitlab-rake "gitlab:uploads:migrate[FileUploader, MergeRequest]"
2020-04-08 14:13:33 +05:30
2020-05-24 23:13:21 +05:30
# Design Management design thumbnails
2020-04-08 14:13:33 +05:30
gitlab-rake "gitlab:uploads:migrate[DesignManagement::DesignV432x230Uploader, DesignManagement::Action, :image_v432x230]"
2018-05-09 12:01:36 +05:30
```
**Source Installation**
Use `RAILS_ENV=production` for every task.
2020-03-13 15:44:24 +05:30
```shell
2018-05-09 12:01:36 +05:30
# sudo -u git -H bundle exec rake gitlab:uploads:migrate
# Avatars
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AvatarUploader, Project, :avatar]"
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AvatarUploader, Group, :avatar]"
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AvatarUploader, User, :avatar]"
# Attachments
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AttachmentUploader, Note, :attachment]"
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :logo]"
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :header_logo]"
2018-12-05 23:21:45 +05:30
# Favicon
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FaviconUploader, Appearance, :favicon]"
2018-05-09 12:01:36 +05:30
# Markdown
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FileUploader, Project]"
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[PersonalFileUploader, Snippet]"
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[NamespaceFileUploader, Snippet]"
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FileUploader, MergeRequest]"
2020-05-24 23:13:21 +05:30
# Design Management design thumbnails
2020-04-08 14:13:33 +05:30
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[DesignManagement::DesignV432x230Uploader, DesignManagement::Action]"
2018-05-09 12:01:36 +05:30
```
2019-10-12 21:52:04 +05:30
2020-05-24 23:13:21 +05:30
## Migrate to local storage
2019-12-21 20:55:43 +05:30
2020-05-24 23:13:21 +05:30
If you need to disable [object storage](../../object_storage.md) for any reason, you must first
migrate your data out of object storage and back into your local storage.
2019-12-21 20:55:43 +05:30
2021-02-22 17:27:13 +05:30
WARNING:
2020-01-01 13:55:28 +05:30
**Extended downtime is required** so no new files are created in object storage during
2021-10-27 15:23:28 +05:30
the migration. A configuration setting to allow migrating
from object storage to local files with only a brief moment of downtime for configuration changes
is tracked [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30979).
2019-12-21 20:55:43 +05:30
2020-04-22 19:07:51 +05:30
### All-in-one Rake task
2019-12-21 20:55:43 +05:30
2020-05-24 23:13:21 +05:30
GitLab provides a wrapper Rake task that migrates all uploaded files (for example, avatars, logos,
attachments, and favicon) to local storage in one step. The wrapper task invokes individual Rake
tasks to migrate files falling under each of these categories one by one.
For details on these Rake tasks, refer to [Individual Rake tasks](#individual-rake-tasks),
2019-12-21 20:55:43 +05:30
keeping in mind the task name in this case is `gitlab:uploads:migrate_to_local`.
2020-05-24 23:13:21 +05:30
To migrate uploads from object storage to local storage:
2019-12-21 20:55:43 +05:30
2020-10-24 23:57:45 +05:30
1. Disable both `direct_upload` and `background_upload` under `uploads` settings in `gitlab.rb`:
```ruby
gitlab_rails['uploads_object_store_direct_upload'] = false
gitlab_rails['uploads_object_store_background_upload'] = false
```
Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure).
2020-05-24 23:13:21 +05:30
1. Run the Rake task:
2019-12-21 20:55:43 +05:30
2020-05-24 23:13:21 +05:30
**Omnibus Installation**
2019-12-21 20:55:43 +05:30
2020-05-24 23:13:21 +05:30
```shell
gitlab-rake "gitlab:uploads:migrate_to_local:all"
```
**Source Installation**
```shell
sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:migrate_to_local:all
```
2019-12-21 20:55:43 +05:30
2020-05-24 23:13:21 +05:30
After running the Rake task, you can disable object storage by undoing the changes described
2020-11-24 15:15:51 +05:30
in the instructions to [configure object storage](../../uploads.md#using-object-storage).