355 lines
14 KiB
Markdown
355 lines
14 KiB
Markdown
---
|
|
stage: Package
|
|
group: Package
|
|
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
|
|
---
|
|
|
|
# Dependency Proxy **(FREE)**
|
|
|
|
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7934) in GitLab 11.11.
|
|
> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/273655) from GitLab Premium to GitLab Free in 13.6.
|
|
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11582) support for private groups in GitLab 13.7.
|
|
> - Anonymous access to images in public groups is no longer available starting in GitLab 13.7.
|
|
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/290944) support for pull-by-digest and Docker version 20.x in GitLab 13.10.
|
|
|
|
The GitLab Dependency Proxy is a local proxy you can use for your frequently-accessed
|
|
upstream images.
|
|
|
|
In the case of CI/CD, the Dependency Proxy receives a request and returns the
|
|
upstream image from a registry, acting as a pull-through cache.
|
|
|
|
## Prerequisites
|
|
|
|
- The Dependency Proxy is enabled by default but can be [turned off by an administrator](../../../administration/packages/dependency_proxy.md).
|
|
|
|
### Supported images and packages
|
|
|
|
The following images and packages are supported.
|
|
|
|
| Image/Package | GitLab version |
|
|
| ---------------- | -------------- |
|
|
| Docker | 11.11+ |
|
|
|
|
For a list of planned additions, view the
|
|
[direction page](https://about.gitlab.com/direction/package/#dependency-proxy).
|
|
|
|
## Enable or disable the Dependency Proxy for a group
|
|
|
|
To enable or disable the Dependency Proxy for a group:
|
|
|
|
1. Go to your group's **Settings > Packages & Registries**.
|
|
1. Expand the **Dependency Proxy** section.
|
|
1. To enable the proxy, turn on **Enable Proxy**. To disable it, turn the toggle off.
|
|
|
|
## View the Dependency Proxy
|
|
|
|
To view the Dependency Proxy:
|
|
|
|
- Go to your group's **Packages & Registries > Dependency Proxy**.
|
|
|
|
The Dependency Proxy is not available for projects.
|
|
|
|
## Use the Dependency Proxy for Docker images
|
|
|
|
You can use GitLab as a source for your Docker images.
|
|
|
|
Prerequisites:
|
|
|
|
- Your images must be stored on [Docker Hub](https://hub.docker.com/).
|
|
|
|
### Authenticate with the Dependency Proxy
|
|
|
|
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11582) in GitLab 13.7.
|
|
> - It's [deployed behind a feature flag](../../feature_flags.md), enabled by default.
|
|
> - It's enabled on GitLab.com.
|
|
> - It's recommended for production use.
|
|
> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](../../../administration/packages/dependency_proxy.md#disabling-authentication). **(FREE SELF)**
|
|
|
|
WARNING:
|
|
This feature might not be available to you. Check the **version history** note above for details.
|
|
The requirement to authenticate is a breaking change added in 13.7. An [administrator can temporarily
|
|
disable it](../../../administration/packages/dependency_proxy.md#disabling-authentication) if it
|
|
has disrupted your existing Dependency Proxy usage.
|
|
|
|
Because the Dependency Proxy is storing Docker images in a space associated with your group,
|
|
you must authenticate against the Dependency Proxy.
|
|
|
|
Follow the [instructions for using images from a private registry](../../../ci/docker/using_docker_images.md#access-an-image-from-a-private-container-registry),
|
|
but instead of using `registry.example.com:5000`, use your GitLab domain with no port `gitlab.example.com`.
|
|
|
|
For example, to manually log in:
|
|
|
|
```shell
|
|
docker login gitlab.example.com --username my_username --password my_password
|
|
```
|
|
|
|
You can authenticate using:
|
|
|
|
- Your GitLab username and password.
|
|
- A [personal access token](../../../user/profile/personal_access_tokens.md) with the scope set to `read_registry` and `write_registry`.
|
|
- A [group deploy token](../../../user/project/deploy_tokens/index.md#group-deploy-token) with the scope set to `read_registry` and `write_registry`.
|
|
|
|
#### SAML SSO
|
|
|
|
When [SSO enforcement](../../group/saml_sso/index.md#sso-enforcement)
|
|
is enabled, users must be signed-in through SSO before they can pull images through the Dependency
|
|
Proxy.
|
|
|
|
#### Authenticate within CI/CD
|
|
|
|
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/280582) in GitLab 13.7.
|
|
> - Automatic runner authentication, when using the Dependency Proxy to pull the image for the job, was [added](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27302) in GitLab 13.9.
|
|
> - The prefix for group names containing uppercase letters was [fixed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/54559) in GitLab 13.10.
|
|
|
|
Runners log in to the Dependency Proxy automatically. To pull through
|
|
the Dependency Proxy, use one of the [predefined variables](../../../ci/variables/predefined_variables.md):
|
|
|
|
- `CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX` pulls through the top-level group.
|
|
- `CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX` pulls through the subgroup, or direct group the
|
|
project exists in.
|
|
|
|
Example pulling the latest alpine image:
|
|
|
|
```yaml
|
|
# .gitlab-ci.yml
|
|
image: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/alpine:latest
|
|
```
|
|
|
|
There are other additional predefined CI/CD variables you can also use:
|
|
|
|
- `CI_DEPENDENCY_PROXY_USER`: A CI/CD user for logging in to the Dependency Proxy.
|
|
- `CI_DEPENDENCY_PROXY_PASSWORD`: A CI/CD password for logging in to the Dependency Proxy.
|
|
- `CI_DEPENDENCY_PROXY_SERVER`: The server for logging in to the Dependency Proxy.
|
|
- `CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX`: the image prefix for pulling images through the
|
|
dependency proxy from the top-level group.
|
|
- `CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX`: the image prefix for pulling images through the
|
|
dependency proxy from the direct group or subgroup that the project belongs to.
|
|
|
|
`CI_DEPENDENCY_PROXY_SERVER` and `CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX`
|
|
include the server port. If you explicitly include the Dependency Proxy
|
|
path, the port must be included, unless you have logged into the Dependency
|
|
Proxy manually without including the port:
|
|
|
|
```shell
|
|
docker pull gitlab.example.com:443/my-group/dependency_proxy/containers/alpine:latest
|
|
```
|
|
|
|
Example when using the Dependency Proxy to build an image:
|
|
|
|
```plaintext
|
|
# Dockerfile
|
|
FROM gitlab.example.com:443/my-group/dependency_proxy/containers/alpine:latest
|
|
```
|
|
|
|
```yaml
|
|
# .gitlab-ci.yml
|
|
image: docker:19.03.12
|
|
|
|
variables:
|
|
DOCKER_HOST: tcp://docker:2375
|
|
DOCKER_TLS_CERTDIR: ""
|
|
|
|
services:
|
|
- docker:19.03.12-dind
|
|
|
|
build:
|
|
image: docker:19.03.12
|
|
before_script:
|
|
- docker login -u $CI_DEPENDENCY_PROXY_USER -p $CI_DEPENDENCY_PROXY_PASSWORD $CI_DEPENDENCY_PROXY_SERVER
|
|
script:
|
|
- docker build -t test .
|
|
```
|
|
|
|
You can also use [custom CI/CD variables](../../../ci/variables/index.md#custom-cicd-variables) to store and access your personal access token or deploy token.
|
|
|
|
### Store a Docker image in Dependency Proxy cache
|
|
|
|
To store a Docker image in Dependency Proxy storage:
|
|
|
|
1. Go to your group's **Packages & Registries > Dependency Proxy**.
|
|
1. Copy the **Dependency Proxy URL**.
|
|
1. Use one of these commands. In these examples, the image is `alpine:latest`.
|
|
1. You can also pull images by digest to specify exactly which version of an image to pull.
|
|
|
|
- Pull an image by tag by adding the image to your [`.gitlab-ci.yml`](../../../ci/yaml/index.md#image) file:
|
|
|
|
```shell
|
|
image: gitlab.example.com/groupname/dependency_proxy/containers/alpine:latest
|
|
```
|
|
|
|
- Pull an image by digest by adding the image to your [`.gitlab-ci.yml`](../../../ci/yaml/index.md#image) file:
|
|
|
|
```shell
|
|
image: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/alpine@sha256:c9375e662992791e3f39e919b26f510e5254b42792519c180aad254e6b38f4dc
|
|
```
|
|
|
|
- Manually pull the Docker image:
|
|
|
|
```shell
|
|
docker pull gitlab.example.com/groupname/dependency_proxy/containers/alpine:latest
|
|
```
|
|
|
|
- Add the URL to a `Dockerfile`:
|
|
|
|
```shell
|
|
FROM gitlab.example.com/groupname/dependency_proxy/containers/alpine:latest
|
|
```
|
|
|
|
GitLab pulls the Docker image from Docker Hub and caches the blobs
|
|
on the GitLab server. The next time you pull the same image, GitLab gets the latest
|
|
information about the image from Docker Hub, but serves the existing blobs
|
|
from the GitLab server.
|
|
|
|
## Clear the Dependency Proxy cache
|
|
|
|
Blobs are kept forever on the GitLab server, and there is no hard limit on how much data can be
|
|
stored.
|
|
|
|
### Using the API to clear the cache
|
|
|
|
To reclaim disk space used by image blobs that are no longer needed, use
|
|
the [Dependency Proxy API](../../../api/dependency_proxy.md) to clear the entire
|
|
cache.
|
|
|
|
If you clear the cache, the next time a pipeline runs it must pull an image or tag from Docker Hub.
|
|
|
|
### Cleanup policies
|
|
|
|
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294187) in GitLab 14.4.
|
|
|
|
The cleanup policy is a scheduled job you can use to clear cached images that are no longer used,
|
|
freeing up additional storage space. The policies use time-to-live (TTL) logic:
|
|
|
|
- The number of days is configured.
|
|
- All cached dependency proxy files that have not been pulled in that many days are deleted.
|
|
|
|
Use the [GraphQL API](../../../api/graphql/reference/index.md#mutationupdatedependencyproxyimagettlgrouppolicy)
|
|
to enable and configure cleanup policies:
|
|
|
|
```graphql
|
|
mutation {
|
|
updateDependencyProxyImageTtlGroupPolicy(input:
|
|
{
|
|
groupPath: "<your-full-group-path>",
|
|
enabled: true,
|
|
ttl: 90
|
|
}
|
|
) {
|
|
dependencyProxyImageTtlPolicy {
|
|
enabled
|
|
ttl
|
|
}
|
|
errors
|
|
}
|
|
}
|
|
```
|
|
|
|
See the [Getting started with GraphQL](../../../api/graphql/getting_started.md)
|
|
guide to learn how to make GraphQL queries. Support for enabling and configuring cleanup policies in
|
|
the UI is tracked in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/340777).
|
|
|
|
When the policy is initially enabled, the default TTL setting is 90 days. Once enabled, stale
|
|
dependency proxy files are queued for deletion each day. Deletion may not occur right away due to
|
|
processing time. If the image is pulled after the cached files are marked as expired, the expired
|
|
files are ignored and new files are downloaded and cached from the external registry.
|
|
|
|
## Docker Hub rate limits and the Dependency Proxy
|
|
|
|
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241639) in GitLab 13.7.
|
|
|
|
<i class="fa fa-youtube-play youtube" aria-hidden="true"></i>
|
|
Watch how to [use the Dependency Proxy to help avoid Docker Hub rate limits](https://youtu.be/Nc4nUo7Pq08).
|
|
|
|
In November 2020, Docker introduced
|
|
[rate limits on pull requests from Docker Hub](https://docs.docker.com/docker-hub/download-rate-limit/).
|
|
If your GitLab [CI/CD configuration](../../../ci/index.md) uses
|
|
an image from Docker Hub, each time a job runs, it may count as a pull request.
|
|
To help get around this limit, you can pull your image from the Dependency Proxy cache instead.
|
|
|
|
When you pull an image (by using a command like `docker pull` or, in a `.gitlab-ci.yml`
|
|
file, `image: foo:latest`), the Docker client makes a collection of requests:
|
|
|
|
1. The image manifest is requested. The manifest contains information about
|
|
how to build the image.
|
|
1. Using the manifest, the Docker client requests a collection of layers, also
|
|
known as blobs, one at a time.
|
|
|
|
The Docker Hub rate limit is based on the number of GET requests for the manifest. The Dependency Proxy
|
|
caches both the manifest and blobs for a given image, so when you request it again,
|
|
Docker Hub does not have to be contacted.
|
|
|
|
### How does GitLab know if a cached tagged image is stale?
|
|
|
|
If you are using an image tag like `alpine:latest`, the image changes
|
|
over time. Each time it changes, the manifest contains different information about which
|
|
blobs to request. The Dependency Proxy does not pull a new image each time the
|
|
manifest changes; it checks only when the manifest becomes stale.
|
|
|
|
Docker does not count HEAD requests for the image manifest towards the rate limit.
|
|
You can make a HEAD request for `alpine:latest`, view the digest (checksum)
|
|
value returned in the header, and determine if a manifest has changed.
|
|
|
|
The Dependency Proxy starts all requests with a HEAD request. If the manifest
|
|
has become stale, only then is a new image pulled.
|
|
|
|
For example, if your pipeline pulls `node:latest` every five
|
|
minutes, the Dependency Proxy caches the entire image and only updates it if
|
|
`node:latest` changes. So instead of having 360 requests for the image in six hours
|
|
(which exceeds the Docker Hub rate limit), you only have one pull request, unless
|
|
the manifest changed during that time.
|
|
|
|
### Check your Docker Hub rate limit
|
|
|
|
If you are curious about how many requests to Docker Hub you have made and how
|
|
many remain, you can run these commands from your runner, or even in a CI/CD
|
|
script:
|
|
|
|
```shell
|
|
# Note, you must have jq installed to run this command
|
|
TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq --raw-output .token) && curl --head --header "Authorization: Bearer $TOKEN" "https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest" 2>&1 | grep --ignore-case RateLimit
|
|
...
|
|
```
|
|
|
|
The output is something like:
|
|
|
|
```shell
|
|
RateLimit-Limit: 100;w=21600
|
|
RateLimit-Remaining: 98;w=21600
|
|
```
|
|
|
|
This example shows the total limit of 100 pulls in six hours, with 98 pulls remaining.
|
|
|
|
#### Check the rate limit in a CI/CD job
|
|
|
|
This example shows a GitLab CI/CD job that uses an image with `jq` and `curl` installed:
|
|
|
|
```yaml
|
|
hub_docker_quota_check:
|
|
stage: build
|
|
image: alpine:latest
|
|
tags:
|
|
- <optional_runner_tag>
|
|
before_script: apk add curl jq
|
|
script:
|
|
- |
|
|
TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq --raw-output .token) && curl --head --header "Authorization: Bearer $TOKEN" "https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest" 2>&1
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Dependency Proxy Connection Failure
|
|
|
|
If a service alias is not set the `docker:19.03.12` image is unable to find the
|
|
`dind` service, and an error like the following is thrown:
|
|
|
|
```plaintext
|
|
error during connect: Get http://docker:2376/v1.39/info: dial tcp: lookup docker on 192.168.0.1:53: no such host
|
|
```
|
|
|
|
This can be resolved by setting a service alias for the Docker service:
|
|
|
|
```yaml
|
|
services:
|
|
- name: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/docker:18.09.7-dind
|
|
alias: docker
|
|
```
|