169 lines
8.2 KiB
Markdown
169 lines
8.2 KiB
Markdown
---
|
|
stage: enablement
|
|
group: pods
|
|
comments: false
|
|
description: 'Pods: CI Runners'
|
|
---
|
|
|
|
This document is a work-in-progress and represents a very early state of the
|
|
Pods design. Significant aspects are not documented, though we expect to add
|
|
them in the future. This is one possible architecture for Pods, and we intend to
|
|
contrast this with alternatives before deciding which approach to implement.
|
|
This documentation will be kept even if we decide not to implement this so that
|
|
we can document the reasons for not choosing this approach.
|
|
|
|
# Pods: CI Runners
|
|
|
|
GitLab in order to execute CI jobs [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/),
|
|
very often managed by customer in their infrastructure.
|
|
|
|
All CI jobs created as part of CI pipeline are run in a context of project
|
|
it poses a challenge how to manage GitLab Runners.
|
|
|
|
## 1. Definition
|
|
|
|
There are 3 different types of runners:
|
|
|
|
- instance-wide: runners that are registered globally with specific tags (selection criteria)
|
|
- group runners: runners that execute jobs from a given top-level group or subprojects of that group
|
|
- project runners: runners that execute jobs from projects or many projects: some runners might
|
|
have projects assigned from projects in different top-level groups.
|
|
|
|
This alongside with existing data structure where `ci_runners` is a table describing
|
|
all types of runners poses a challenge how the `ci_runners` should be managed in a Pods environment.
|
|
|
|
## 2. Data flow
|
|
|
|
GitLab Runners use a set of globally scoped endpoints to:
|
|
|
|
- registration of a new runner via registration token `https://gitlab.com/api/v4/runners`
|
|
([subject for removal](../runner_tokens/index.md)) (`registration token`)
|
|
- requests jobs via an authenticated `https://gitlab.com/api/v4/jobs/request` endpoint (`runner token`)
|
|
- upload job status via `https://gitlab.com/api/v4/jobs/:job_id` (`build token`)
|
|
- upload trace via `https://gitlab.com/api/v4/jobs/:job_id/trace` (`build token`)
|
|
- download and upload artifacts via `https://gitlab.com/api/v4/jobs/:job_id/artifacts` (`build token`)
|
|
|
|
Currently three types of authentication tokens are used:
|
|
|
|
- runner registration token ([subject for removal](../runner_tokens/index.md))
|
|
- runner token representing an registered runner in a system with specific configuration (`tags`, `locked`, etc.)
|
|
- build token representing an ephemeral token giving a limited access to updating a specific
|
|
job, uploading artifacts, downloading dependent artifacts, downloading and uploading
|
|
container registry images
|
|
|
|
Each of those endpoints do receive an authentication token via header (`JOB-TOKEN` for `/trace`)
|
|
or body parameter (`token` all other endpoints).
|
|
|
|
Since the CI pipeline would be created in a context of a specific Pod it would be required
|
|
that pick of a build would have to be processed by that particular Pod. This requires
|
|
that build picking depending on a solution would have to be either:
|
|
|
|
- routed to correct Pod for a first time
|
|
- be made to be two phase: request build from global pool, claim build on a specific Pod using a Pod specific URL
|
|
|
|
## 3. Proposal
|
|
|
|
This section describes various proposals. Reader should consider that those
|
|
proposals do describe solutions for different problems. Many or some aspects
|
|
of those proposals might be the solution to the stated problem.
|
|
|
|
### 3.1. Authentication tokens
|
|
|
|
Even though the paths for CI Runners are not routable they can be made routable with
|
|
those two possible solutions:
|
|
|
|
- The `https://gitlab.com/api/v4/jobs/request` uses a long polling mechanism with
|
|
a ticketing mechanism (based on `X-GitLab-Last-Update` header). Runner when first
|
|
starts sends a request to GitLab to which GitLab responds with either a build to pick
|
|
by runner. This value is completely controlled by GitLab. This allows GitLab
|
|
to use JWT or any other means to encode `pod` identifier that could be easily
|
|
decodable by Router.
|
|
- The majority of communication (in terms of volume) is using `build token` making it
|
|
the easiest target to change since GitLab is sole owner of the token that Runner later
|
|
uses for specific job. There were prior discussions about not storing `build token`
|
|
but rather using `JWT` token with defined scopes. Such token could encode the `pod`
|
|
to which router could easily route all requests.
|
|
|
|
### 3.2. Request body
|
|
|
|
- The most of used endpoints pass authentication token in request body. It might be desired
|
|
to use HTTP Headers as an easier way to access this information by Router without
|
|
a need to proxy requests.
|
|
|
|
### 3.3. Instance-wide are Pod local
|
|
|
|
We can pick a design where all runners are always registered and local to a given Pod:
|
|
|
|
- Each Pod has it's own set of instance-wide runners that are updated at it's own pace
|
|
- The project runners can only be linked to projects from the same organization
|
|
creating strong isolation.
|
|
- In this model the `ci_runners` table is local to the Pod.
|
|
- In this model we would require the above endpoints to be scoped to a Pod in some way
|
|
or made routable. It might be via prefixing them, adding additional Pod parameter,
|
|
or providing much more robust way to decode runner token and match it to Pod.
|
|
- If routable token is used, we could move away from cryptographic random stored in
|
|
database to rather prefer to use JWT tokens that would encode
|
|
- The Admin Area showing registered Runners would have to be scoped to a Pod
|
|
|
|
This model might be desired since it provides strong isolation guarantees.
|
|
This model does significantly increase maintenance overhead since each Pod is managed
|
|
separately.
|
|
|
|
This model may require adjustments to runner tags feature so that projects have consistent runner experience across pods.
|
|
|
|
### 3.4. Instance-wide are cluster-wide
|
|
|
|
Contrary to proposal where all runners are Pod local, we can consider that runners
|
|
are global, or just instance-wide runners are global.
|
|
|
|
However, this requires significant overhaul of system and to change the following aspects:
|
|
|
|
- `ci_runners` table would likely have to be split decomposed into `ci_instance_runners`, ...
|
|
- all interfaces would have to be adopted to use correct table
|
|
- build queuing would have to be reworked to be two phase where each Pod would know of all pending
|
|
and running builds, but the actual claim of a build would happen against a Pod containing data
|
|
- likely `ci_pending_builds` and `ci_running_builds` would have to be made `cluster-wide` tables
|
|
increasing likelihood of creating hotspots in a system related to CI queueing
|
|
|
|
This model makes it complex to implement from engineering side. Does make some data being shared
|
|
between Pods. Creates hotspots / scalability issues in a system (ex. during abuse) that
|
|
might impact experience of organizations on other Pods.
|
|
|
|
### 3.5. GitLab CI Daemon
|
|
|
|
Another potential solution to explore is to have a dedicated service responsible for builds queueing
|
|
owning it's database and working in a model of either sharded or podded service. There were prior
|
|
discussions about [CI/CD Daemon](https://gitlab.com/gitlab-org/gitlab/-/issues/19435).
|
|
|
|
If the service would be sharded:
|
|
|
|
- depending on a model if runners are cluster-wide or pod-local this service would have to fetch
|
|
data from all Pods
|
|
- if the sharded service would be used we could adapt a model of either sharing database containing
|
|
`ci_pending_builds/ci_running_builds` with the service
|
|
- if the sharded service would be used we could consider a push model where each Pod pushes to CI/CD Daemon
|
|
builds that should be picked by Runner
|
|
- the sharded service would be aware which Pod is responsible for processing the given build and could
|
|
route processing requests to designated Pod
|
|
|
|
If the service would be podded:
|
|
|
|
- all expectations of routable endpoints are still valid
|
|
|
|
In general usage of CI Daemon does not help significantly with the stated problem. However, this offers
|
|
a few upsides related to more efficient processing and decoupling model: push model and it opens a way
|
|
to offer stateful communication with GitLab Runners (ex. gRPC or Websockets).
|
|
|
|
## 4. Evaluation
|
|
|
|
Considering all solutions it appears that solution giving the most promise is:
|
|
|
|
- use "instance-wide are Pod local"
|
|
- refine endpoints to have routable identities (either via specific paths, or better tokens)
|
|
|
|
Other potential upsides is to get rid of `ci_builds.token` and rather use a `JWT token`
|
|
that can much better and easier encode wider set of scopes allowed by CI runner.
|
|
|
|
## 4.1. Pros
|
|
|
|
## 4.2. Cons
|