debian-mirror-gitlab/doc/development/file_storage.md

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

157 lines
9.1 KiB
Markdown
Raw Normal View History

2021-01-29 00:20:46 +05:30
---
stage: none
group: unassigned
2021-02-22 17:27:13 +05:30
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
2021-01-29 00:20:46 +05:30
---
2018-03-17 18:26:18 +05:30
# File Storage in GitLab
2020-04-22 19:07:51 +05:30
We use the [CarrierWave](https://github.com/carrierwaveuploader/carrierwave) gem to handle file upload, store and retrieval.
2018-03-17 18:26:18 +05:30
2022-05-07 20:08:51 +05:30
File uploads should be accelerated by workhorse, for details please refer to [uploads development documentation](uploads/index.md).
2019-12-04 20:38:33 +05:30
2018-03-17 18:26:18 +05:30
There are many places where file uploading is used, according to contexts:
2019-03-02 22:35:43 +05:30
- System
2018-03-17 18:26:18 +05:30
- Instance Logo (logo visible in sign in/sign up pages)
- Header Logo (one displayed in the navigation bar)
2019-03-02 22:35:43 +05:30
- Group
2018-03-17 18:26:18 +05:30
- Group avatars
2019-03-02 22:35:43 +05:30
- User
2018-03-17 18:26:18 +05:30
- User avatars
- User snippet attachments
2019-03-02 22:35:43 +05:30
- Project
2018-03-17 18:26:18 +05:30
- Project avatars
- Issues/MR/Notes Markdown attachments
- Issues/MR/Notes Legacy Markdown attachments
- CI Artifacts (archive, metadata, trace)
- LFS Objects
2019-03-02 22:35:43 +05:30
- Merge request diffs
2020-05-24 23:13:21 +05:30
- Design Management design thumbnails
2021-11-18 22:05:49 +05:30
- Topic
- Topic avatars
2018-03-17 18:26:18 +05:30
## Disk storage
GitLab started saving everything on local disk. While directory location changed from previous versions,
they are still not 100% standardized. You can see them below:
| Description | In DB? | Relative path (from CarrierWave.root) | Uploader class | model_type |
| ------------------------------------- | ------ | ----------------------------------------------------------- | ---------------------- | ---------- |
2020-05-24 23:13:21 +05:30
| Instance logo | yes | `uploads/-/system/appearance/logo/:id/:filename` | `AttachmentUploader` | Appearance |
| Header logo | yes | `uploads/-/system/appearance/header_logo/:id/:filename` | `AttachmentUploader` | Appearance |
| Group avatars | yes | `uploads/-/system/group/avatar/:id/:filename` | `AvatarUploader` | Group |
| User avatars | yes | `uploads/-/system/user/avatar/:id/:filename` | `AvatarUploader` | User |
| User snippet attachments | yes | `uploads/-/system/personal_snippet/:id/:random_hex/:filename` | `PersonalFileUploader` | Snippet |
| Project avatars | yes | `uploads/-/system/project/avatar/:id/:filename` | `AvatarUploader` | Project |
2021-11-18 22:05:49 +05:30
| Topic avatars | yes | `uploads/-/system/projects/topic/avatar/:id/:filename` | `AvatarUploader` | Topic |
2020-05-24 23:13:21 +05:30
| Issues/MR/Notes Markdown attachments | yes | `uploads/:project_path_with_namespace/:random_hex/:filename` | `FileUploader` | Project |
| Issues/MR/Notes Legacy Markdown attachments | no | `uploads/-/system/note/attachment/:id/:filename` | `AttachmentUploader` | Note |
| Design Management design thumbnails | yes | `uploads/-/system/design_management/action/image_v432x230/:id/:filename` | `DesignManagement::DesignV432x230Uploader` | DesignManagement::Action |
| CI Artifacts (CE) | yes | `shared/artifacts/:disk_hash[0..1]/:disk_hash[2..3]/:disk_hash/:year_:month_:date/:job_id/:job_artifact_id` (`:disk_hash` is SHA256 digest of `project_id`) | `JobArtifactUploader` | Ci::JobArtifact |
| LFS Objects (CE) | yes | `shared/lfs-objects/:hex/:hex/:object_hash` | `LfsObjectUploader` | LfsObject |
| External merge request diffs | yes | `shared/external-diffs/merge_request_diffs/mr-:parent_id/diff-:id` | `ExternalDiffUploader` | MergeRequestDiff |
2021-02-22 17:27:13 +05:30
| Issuable metric images | yes | `uploads/-/system/issuable_metric_image/file/:id/:filename` | `IssuableMetricImageUploader` | IssuableMetricImage |
2018-03-17 18:26:18 +05:30
CI Artifacts and LFS Objects behave differently in CE and EE. In CE they inherit the `GitlabUploader`
while in EE they inherit the `ObjectStorage` and store files in and S3 API compatible object store.
2020-04-22 19:07:51 +05:30
In the case of Issues/MR/Notes Markdown attachments, there is a different approach using the [Hashed Storage](../administration/repository_storage_types.md) layout,
2018-03-17 18:26:18 +05:30
instead of basing the path into a mutable variable `:project_path_with_namespace`, it's possible to use the
hash of the project ID instead, if project migrates to the new approach (introduced in 10.2).
2021-01-29 00:20:46 +05:30
We provide an [all-in-one Rake task](../administration/raketasks/uploads/migrate.md)
to migrate all uploads to object storage in one go. If a new Uploader class or model
type is introduced, make sure you add a Rake task invocation corresponding to it to the
2021-09-04 01:27:46 +05:30
[category list](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/tasks/gitlab/uploads/migrate.rake).
2018-12-05 23:21:45 +05:30
2018-03-17 18:26:18 +05:30
### Path segments
2019-03-02 22:35:43 +05:30
Files are stored at multiple locations and use different path schemes.
2018-03-17 18:26:18 +05:30
All the `GitlabUploader` derived classes should comply with this path segment schema:
2020-04-22 19:07:51 +05:30
```plaintext
2018-03-17 18:26:18 +05:30
| GitlabUploader
| ----------------------- + ------------------------- + --------------------------------- + -------------------------------- |
| `<gitlab_root>/public/` | `uploads/-/system/` | `user/avatar/:id/` | `:filename` |
| ----------------------- + ------------------------- + --------------------------------- + -------------------------------- |
| `CarrierWave.root` | `GitlabUploader.base_dir` | `GitlabUploader#dynamic_segment` | `CarrierWave::Uploader#filename` |
2019-03-02 22:35:43 +05:30
| | `CarrierWave::Uploader#store_dir` | |
2018-03-17 18:26:18 +05:30
| FileUploader
| ----------------------- + ------------------------- + --------------------------------- + -------------------------------- |
| `<gitlab_root>/shared/` | `artifacts/` | `:year_:month/:id` | `:filename` |
| `<gitlab_root>/shared/` | `snippets/` | `:secret/` | `:filename` |
| ----------------------- + ------------------------- + --------------------------------- + -------------------------------- |
| `CarrierWave.root` | `GitlabUploader.base_dir` | `GitlabUploader#dynamic_segment` | `CarrierWave::Uploader#filename` |
2019-03-02 22:35:43 +05:30
| | `CarrierWave::Uploader#store_dir` | |
2018-03-17 18:26:18 +05:30
| | | `FileUploader#upload_path |
| ObjectStore::Concern (store = remote)
| ----------------------- + ------------------------- + ----------------------------------- + -------------------------------- |
| `<bucket_name>` | <ignored> | `user/avatar/:id/` | `:filename` |
| ----------------------- + ------------------------- + ----------------------------------- + -------------------------------- |
| `#fog_dir` | `GitlabUploader.base_dir` | `GitlabUploader#dynamic_segment` | `CarrierWave::Uploader#filename` |
2019-03-02 22:35:43 +05:30
| | | `ObjectStorage::Concern#store_dir` | |
2018-03-17 18:26:18 +05:30
| | | `ObjectStorage::Concern#upload_path |
```
2021-02-22 17:27:13 +05:30
The `RecordsUploads::Concern` concern creates an `Upload` entry for every file stored by a `GitlabUploader` persisting the dynamic parts of the path using
2018-03-17 18:26:18 +05:30
`GitlabUploader#dynamic_path`. You may then use the `Upload#build_uploader` method to manipulate the file.
## Object Storage
By including the `ObjectStorage::Concern` in the `GitlabUploader` derived class, you may enable the object storage for this uploader. To enable the object storage
in your uploader, you need to either 1) include `RecordsUpload::Concern` and prepend `ObjectStorage::Extension::RecordsUploads` or 2) mount the uploader and create a new field named `<mount>_store`.
2018-10-15 14:42:47 +05:30
The `CarrierWave::Uploader#store_dir` is overridden to
2018-03-17 18:26:18 +05:30
2019-09-30 21:07:59 +05:30
- `GitlabUploader.base_dir` + `GitlabUploader.dynamic_segment` when the store is LOCAL
- `GitlabUploader.dynamic_segment` when the store is REMOTE (the bucket name is used to namespace)
2018-03-17 18:26:18 +05:30
### Using `ObjectStorage::Extension::RecordsUploads`
2021-02-22 17:27:13 +05:30
This concern includes `RecordsUploads::Concern` if not already included.
2018-03-17 18:26:18 +05:30
2021-02-22 17:27:13 +05:30
The `ObjectStorage::Concern` uploader searches for the matching `Upload` to select the correct object store. The `Upload` is mapped using `#store_dirs + identifier` for each store (LOCAL/REMOTE).
2018-03-17 18:26:18 +05:30
```ruby
class SongUploader < GitlabUploader
include RecordsUploads::Concern
include ObjectStorage::Concern
prepend ObjectStorage::Extension::RecordsUploads
...
end
class Thing < ActiveRecord::Base
mount :theme, SongUploader # we have a great theme song!
...
end
```
### Using a mounted uploader
2021-02-22 17:27:13 +05:30
The `ObjectStorage::Concern` queries the `model.<mount>_store` attribute to select the correct object store.
2018-03-17 18:26:18 +05:30
This column must be present in the model schema.
```ruby
class SongUploader < GitlabUploader
include ObjectStorage::Concern
...
end
class Thing < ActiveRecord::Base
attr_reader :theme_store # this is an ActiveRecord attribute
mount :theme, SongUploader # we have a great theme song!
def theme_store
super || ObjectStorage::Store::LOCAL
end
...
end
```