debian-mirror-gitlab/doc/administration/packages/dependency_proxy.md

273 lines
10 KiB
Markdown
Raw Normal View History

2020-06-23 00:09:42 +05:30
---
stage: Package
2023-01-13 00:05:48 +05:30
group: Container Registry
2022-11-25 23:54:43 +05:30
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
2020-06-23 00:09:42 +05:30
---
2021-04-29 21:17:54 +05:30
# GitLab Dependency Proxy administration **(FREE SELF)**
2019-12-04 20:38:33 +05:30
2021-01-29 00:20:46 +05:30
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7934) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.11.
2021-11-11 11:23:49 +05:30
> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/273655) from GitLab Premium to GitLab Free in 13.6.
2019-12-04 20:38:33 +05:30
2021-01-29 00:20:46 +05:30
GitLab can be used as a dependency proxy for a variety of common package managers.
2019-12-04 20:38:33 +05:30
This is the administration documentation. If you want to learn how to use the
2021-01-03 14:25:43 +05:30
dependency proxies, see the [user guide](../../user/packages/dependency_proxy/index.md).
2019-12-04 20:38:33 +05:30
2022-03-02 08:16:31 +05:30
The GitLab Dependency Proxy:
2019-12-04 20:38:33 +05:30
2022-03-02 08:16:31 +05:30
- Is turned on by default.
- Can be turned off by an administrator.
- Requires the [Puma web server](../operations/puma.md)
to be enabled. Puma is enabled by default in GitLab 13.0 and later.
2019-12-04 20:38:33 +05:30
2022-03-02 08:16:31 +05:30
## Turn off the Dependency Proxy
2019-12-04 20:38:33 +05:30
2022-03-02 08:16:31 +05:30
The Dependency Proxy is enabled by default. If you are an administrator, you
can turn off the Dependency Proxy. To turn off the Dependency Proxy, follow the instructions that
correspond to your GitLab installation:
- [Omnibus GitLab installations](#omnibus-gitlab-installations)
- [Helm chart installations](#helm-chart-installations)
- [Installations from source](#installations-from-source)
### Omnibus GitLab installations
2019-12-04 20:38:33 +05:30
1. Edit `/etc/gitlab/gitlab.rb` and add the following line:
```ruby
2022-03-02 08:16:31 +05:30
gitlab_rails['dependency_proxy_enabled'] = false
2019-12-04 20:38:33 +05:30
```
2022-03-02 08:16:31 +05:30
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure)
for the changes to take effect.
2019-12-04 20:38:33 +05:30
2022-03-02 08:16:31 +05:30
### Helm chart installations
2021-09-04 01:27:46 +05:30
2022-03-02 08:16:31 +05:30
After the installation is complete, update the global `appConfig` to turn off the Dependency Proxy:
2021-09-04 01:27:46 +05:30
2022-03-02 08:16:31 +05:30
```yaml
global:
appConfig:
dependencyProxy:
enabled: false
bucket: gitlab-dependency-proxy
connection: {}
secret:
key:
```
2021-09-04 01:27:46 +05:30
For more information, see [Configure Charts using Globals](https://docs.gitlab.com/charts/charts/globals.html#configure-appconfig-settings).
2022-03-02 08:16:31 +05:30
### Installations from source
2019-12-04 20:38:33 +05:30
2022-03-02 08:16:31 +05:30
1. After the installation is complete, configure the `dependency_proxy` section in
`config/gitlab.yml`. Set `enabled` to `false` to turn off the Dependency Proxy:
2019-12-04 20:38:33 +05:30
```yaml
dependency_proxy:
2022-03-02 08:16:31 +05:30
enabled: false
2019-12-04 20:38:33 +05:30
```
2022-03-02 08:16:31 +05:30
1. [Restart GitLab](../restart_gitlab.md#installations-from-source "How to restart GitLab")
for the changes to take effect.
### Multi-node GitLab installations
2020-04-08 14:13:33 +05:30
2022-03-02 08:16:31 +05:30
Follow the steps for [Omnibus GitLab installations](#omnibus-gitlab-installations)
for each Web and Sidekiq node.
2019-12-04 20:38:33 +05:30
2022-03-02 08:16:31 +05:30
## Turn on the Dependency Proxy
2021-04-17 20:07:23 +05:30
2022-03-02 08:16:31 +05:30
The Dependency Proxy is turned on by default, but can be turned off by an
administrator. To turn on the Dependency Proxy, follow the instructions in
[Turn off the Dependency Proxy](#turn-off-the-dependency-proxy),
but set the `enabled` fields to `true`.
2021-04-17 20:07:23 +05:30
2019-12-04 20:38:33 +05:30
## Changing the storage path
2022-03-02 08:16:31 +05:30
By default, the Dependency Proxy files are stored locally, but you can change the default
2019-12-04 20:38:33 +05:30
local location or even use object storage.
### Changing the local storage path
2022-03-02 08:16:31 +05:30
The Dependency Proxy files for Omnibus GitLab installations are stored under
2019-12-04 20:38:33 +05:30
`/var/opt/gitlab/gitlab-rails/shared/dependency_proxy/` and for source
installations under `shared/dependency_proxy/` (relative to the Git home directory).
To change the local storage path:
**Omnibus GitLab installations**
1. Edit `/etc/gitlab/gitlab.rb` and add the following line:
```ruby
gitlab_rails['dependency_proxy_storage_path'] = "/mnt/dependency_proxy"
```
2021-11-18 22:05:49 +05:30
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
2019-12-04 20:38:33 +05:30
**Installations from source**
1. Edit the `dependency_proxy` section in `config/gitlab.yml`:
```yaml
dependency_proxy:
enabled: true
storage_path: shared/dependency_proxy
```
1. [Restart GitLab](../restart_gitlab.md#installations-from-source "How to restart GitLab") for the changes to take effect.
### Using object storage
Instead of relying on the local storage, you can use an object storage to
2022-03-02 08:16:31 +05:30
store the blobs of the Dependency Proxy.
2020-04-22 19:07:51 +05:30
[Read more about using object storage with GitLab](../object_storage.md).
2019-12-04 20:38:33 +05:30
2021-02-22 17:27:13 +05:30
NOTE:
2020-07-28 23:09:34 +05:30
In GitLab 13.2 and later, we recommend using the
[consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration).
This section describes the earlier configuration format.
2019-12-04 20:38:33 +05:30
**Omnibus GitLab installations**
1. Edit `/etc/gitlab/gitlab.rb` and add the following lines (uncomment where
necessary):
```ruby
gitlab_rails['dependency_proxy_enabled'] = true
gitlab_rails['dependency_proxy_storage_path'] = "/var/opt/gitlab/gitlab-rails/shared/dependency_proxy"
gitlab_rails['dependency_proxy_object_store_enabled'] = true
gitlab_rails['dependency_proxy_object_store_remote_directory'] = "dependency_proxy" # The bucket name.
gitlab_rails['dependency_proxy_object_store_proxy_download'] = false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage.
gitlab_rails['dependency_proxy_object_store_connection'] = {
##
## If the provider is AWS S3, uncomment the following
##
#'provider' => 'AWS',
#'region' => 'eu-west-1',
#'aws_access_key_id' => 'AWS_ACCESS_KEY_ID',
#'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY',
##
## If the provider is other than AWS (an S3-compatible one), uncomment the following
##
#'host' => 's3.amazonaws.com',
#'aws_signature_version' => 4 # For creation of signed URLs. Set to 2 if provider does not support v4.
#'endpoint' => 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces.
#'path_style' => false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'.
}
```
2021-11-18 22:05:49 +05:30
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
2019-12-04 20:38:33 +05:30
**Installations from source**
1. Edit the `dependency_proxy` section in `config/gitlab.yml` (uncomment where necessary):
```yaml
dependency_proxy:
enabled: true
##
## The location where build dependency_proxy are stored (default: shared/dependency_proxy).
##
2021-01-03 14:25:43 +05:30
# storage_path: shared/dependency_proxy
2019-12-04 20:38:33 +05:30
object_store:
enabled: false
2021-01-03 14:25:43 +05:30
remote_directory: dependency_proxy # The bucket name.
# proxy_download: false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage.
2019-12-04 20:38:33 +05:30
connection:
2021-01-03 14:25:43 +05:30
##
## If the provider is AWS S3, use the following
##
provider: AWS
region: us-east-1
aws_access_key_id: AWS_ACCESS_KEY_ID
aws_secret_access_key: AWS_SECRET_ACCESS_KEY
2019-12-04 20:38:33 +05:30
##
2021-01-03 14:25:43 +05:30
## If the provider is other than AWS (an S3-compatible one), comment out the previous 4 lines and use the following instead:
2019-12-04 20:38:33 +05:30
##
2021-01-03 14:25:43 +05:30
# host: 's3.amazonaws.com' # default: s3.amazonaws.com.
# aws_signature_version: 4 # For creation of signed URLs. Set to 2 if provider does not support v4.
# endpoint: 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces.
# path_style: false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'.
2019-12-04 20:38:33 +05:30
```
1. [Restart GitLab](../restart_gitlab.md#installations-from-source "How to restart GitLab") for the changes to take effect.
2021-02-22 17:27:13 +05:30
2022-04-04 11:22:00 +05:30
#### Migrate local Dependency Proxy blobs and manifests to object storage
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79663) in GitLab 14.8.
After [configuring object storage](#using-object-storage),
use the following task to migrate existing Dependency Proxy blobs and manifests from local storage
to remote storage. The processing is done in a background worker and requires no downtime.
For Omnibus GitLab:
```shell
sudo gitlab-rake "gitlab:dependency_proxy:migrate"
```
For installations from source:
```shell
RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:dependency_proxy:migrate
```
2023-01-13 00:05:48 +05:30
You can optionally track progress and verify that all Dependency Proxy blobs and manifests migrated successfully using the
2022-04-04 11:22:00 +05:30
[PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database):
2023-01-13 00:05:48 +05:30
- `sudo gitlab-rails dbconsole` for Omnibus GitLab 14.1 and earlier.
- `sudo gitlab-rails dbconsole --database main` for Omnibus GitLab 14.2 and later.
- `sudo -u git -H psql -d gitlabhq_production` for source-installed instances.
2022-04-04 11:22:00 +05:30
Verify that `objectstg` (where `file_store = '2'`) has the count of all Dependency Proxy blobs and
manifests for each respective query:
```shell
gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM dependency_proxy_blobs;
total | filesystem | objectstg
------+------------+-----------
22 | 0 | 22
gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM dependency_proxy_manifests;
total | filesystem | objectstg
------+------------+-----------
10 | 0 | 10
```
Verify that there are no files on disk in the `dependency_proxy` folder:
```shell
sudo find /var/opt/gitlab/gitlab-rails/shared/dependency_proxy -type f | grep -v tmp | wc -l
```
2022-06-21 17:19:12 +05:30
## Changing the JWT expiration
The Dependency Proxy follows the [Docker v2 token authentication flow](https://docs.docker.com/registry/spec/auth/token/),
issuing the client a JWT to use for the pull requests. The token expiration time is a configurable
using the application setting `container_registry_token_expire_delay`. It can be changed from the
rails console:
```ruby
# update the JWT expiration to 30 minutes
ApplicationSetting.update(container_registry_token_expire_delay: 30)
```
The default expiration and the expiration on GitLab.com is 15 minutes.
## Using the dependency proxy behind a proxy
2022-07-16 23:28:13 +05:30
2022-06-21 17:19:12 +05:30
1. Edit `/etc/gitlab/gitlab.rb` and add the following lines:
```ruby
gitlab_workhorse['env'] = {
"http_proxy" => "http://USERNAME:PASSWORD@example.com:8080",
"https_proxy" => "http://USERNAME:PASSWORD@example.com:8080"
```
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.