- title: "Background upload for object storage"
  announcement_milestone: "14.9"
  announcement_date: "2022-03-22"
  removal_milestone: "15.0"
  removal_date: "2022-05-22"
  breaking_change: true
  reporter: fzimmer
  body: |  # Do not modify this line, instead modify the lines below.
    To reduce the overall complexity and maintenance burden of GitLab's [object storage feature](https://docs.gitlab.com/ee/administration/object_storage.html), support for using `background_upload` has been removed in GitLab 15.0.
    By default [direct upload](https://docs.gitlab.com/ee/development/uploads/index.html#direct-upload) will be used.

    This impacts a subset of object storage providers, including but not limited to:

    - **OpenStack** Customers using OpenStack need to change their configuration to use the S3 API instead of Swift.
    - **RackSpace** Customers using RackSpace-based object storage need to migrate data to a different provider.

    If your object storage provider does not support `background_upload`, please [migrate objects to a supported object storage provider](https://docs.gitlab.com/ee/administration/object_storage.html#migrate-objects-to-a-different-object-storage-provider).

    #### Encrypted S3 buckets

    Additionally, this also breaks the use of [encrypted S3 buckets](https://docs.gitlab.com/ee/administration/object_storage.html#encrypted-s3-buckets) with [storage-specific configuration form](https://docs.gitlab.com/ee/administration/object_storage.html#storage-specific-configuration).

    If your S3 buckets have [SSE-S3 or SSE-KMS encryption enabled](https://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html), please [migrate your configuration to use consolidated object storage form](https://docs.gitlab.com/ee/administration/object_storage.html#transition-to-consolidated-form) before upgrading to GitLab 15.0. Otherwise, you may start getting `ETag mismatch` errors during objects upload.

    #### 403 errors

    If you see 403 errors when uploading to object storage after
    upgrading to GitLab 15.0, check that the [correct permissions](https://docs.gitlab.com/ee/administration/object_storage.html#iam-permissions)
    are assigned to the bucket. Direct upload needs the ability to delete an
    object (example: `s3:DeleteObject`), but background uploads do not.

    #### `remote_directory` with a path prefix

    If the object storage `remote_directory` configuration contains a slash (`/`) after the bucket (example: `gitlab/uploads`), be aware that this [was never officially supported](https://gitlab.com/gitlab-org/gitlab/-/issues/292958).
    Some users found that they could specify a path prefix to the bucket. In direct upload mode, object storage uploads will fail if a slash is present in GitLab 15.0.

    If you have set a prefix, you can use a workaround to revert to background uploads:

    1. Continue to use [storage-specific configuration](https://docs.gitlab.com/ee/administration/object_storage.html#storage-specific-configuration).
    1. In Omnibus GitLab, set the `GITLAB_LEGACY_BACKGROUND_UPLOADS` to re-enable background uploads:

        ```ruby
        gitlab_rails['env'] = { 'GITLAB_LEGACY_BACKGROUND_UPLOADS' => 'artifacts,external_diffs,lfs,uploads,packages,dependency_proxy,terraform_state,pages' }
        ```

    Support for prefixes was restored in GitLab 15.2 via [this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91307).
    Support for setting `GITLAB_LEGACY_BACKGROUND_UPLOADS` will be removed in GitLab 15.4.

  stage: Enablement
  tiers: [Core, Premium, Ultimate]
  issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/26600
  documentation_url: https://docs.gitlab.com/ee/administration/object_storage.html