447 lines
24 KiB
Markdown
447 lines
24 KiB
Markdown
---
|
|
stage: none
|
|
group: unassigned
|
|
comments: false
|
|
description: 'Next Runner Auto-scaling Architecture'
|
|
---
|
|
|
|
# Next Runner Auto-scaling Architecture
|
|
|
|
## Summary
|
|
|
|
GitLab Runner is a core component of GitLab CI/CD. It makes it possible to run
|
|
CI/CD jobs in a reliable and concurrent environment. It has been initially
|
|
introduced by Kamil Trzciński in early 2015 to replace a Ruby version of the
|
|
same service. GitLab Runner written in Go turned out to be easier to use by the
|
|
wider community, it was more efficient and reliable than the previous,
|
|
Ruby-based, version.
|
|
|
|
In February 2016 Kamil Trzciński [implemented an auto-scaling feature](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/53)
|
|
to leverage cloud infrastructure to run many CI/CD jobs in parallel. This
|
|
feature has become a foundation supporting CI/CD adoption on GitLab.com over
|
|
the years, where we now run around 4 million builds per day at peak.
|
|
|
|
During the initial implementation a decision was made to use Docker Machine:
|
|
|
|
> Is easy to use. Is well documented. Is well supported and constantly
|
|
> extended. It supports almost any cloud provider or virtualization
|
|
> infrastructure. We need minimal amount of changes to support Docker Machine:
|
|
> machine enumeration and inspection. We don't need to implement any "cloud
|
|
> specific" features.
|
|
|
|
This design choice was crucial for the GitLab Runner success. Since that time
|
|
the auto-scaling feature has been used by many users and customers and enabled
|
|
rapid growth of CI/CD adoption on GitLab.com.
|
|
|
|
We can not, however, continue using Docker Machine. Work on that project
|
|
[was paused in July 2018](https://github.com/docker/machine/issues/4537) and there
|
|
was no development made since that time (except for some highly important
|
|
security fixes). In 2018, after Docker Machine entered the "maintenance mode",
|
|
we decided to create [our own fork](https://gitlab.com/gitlab-org/ci-cd/docker-machine)
|
|
to be able to keep using this and ship fixes and updates needed for our use case.
|
|
[On September 26th, 2021 the project got archived](https://github.com/docker/docker.github.io/commit/2dc8b49dcbe85686cc7230e17aff8e9944cb47a5)
|
|
and the documentation for it has been removed from the official page. This
|
|
means that the original reason to use Docker Machine is no longer valid too.
|
|
|
|
To keep supporting our customers and the wider community and to improve our SaaS runners
|
|
maintenance we need to design a new mechanism for GitLab Runner auto-scaling. It not only
|
|
needs to support auto-scaling, but it also needs to do that in the way to enable us to
|
|
build on top of it to improve efficiency, reliability and availability.
|
|
|
|
We call this new mechanism the "next GitLab Runner Scaling architecture".
|
|
|
|
_Disclaimer The following contain information related to upcoming products,
|
|
features, and functionality._
|
|
|
|
_It is important to note that the information presented is for informational
|
|
purposes only. Please do not rely on this information for purchasing or
|
|
planning purposes._
|
|
|
|
_As with all projects, the items mentioned in this document and linked pages are
|
|
subject to change or delay. The development, release and timing of any
|
|
products, features, or functionality remain at the sole discretion of GitLab
|
|
Inc._
|
|
|
|
## Continuing building on Docker Machine
|
|
|
|
At this moment one of our core products - GitLab Runner - and one of its most
|
|
important features - ability to auto-scale job execution environments - depends
|
|
on an external product that is abandoned.
|
|
|
|
Docker Machine project itself is also hard to maintain. Its design starts to
|
|
show its age, which makes it hard to bring new features and fixes. A huge
|
|
codebase that it brings with a lack of internal knowledge about it makes it
|
|
hard for our maintainers to support and properly handle incoming feature
|
|
requests and community contributions.
|
|
|
|
Docker Machine and it integrated 20+ drivers for cloud and virtualization
|
|
providers creates also another subset of problems, like:
|
|
|
|
- Each cloud/virtualization environment brings features that come and go
|
|
and we would need to maintain support for them (add new features, fix
|
|
bugs).
|
|
|
|
- We basically need to become experts for each of the virtualization/cloud
|
|
provider to properly support integration with their API,
|
|
|
|
- Every single provider that Docker Machine integrates with has its
|
|
bugs, security releases, vulnerabilities - to maintain the project properly
|
|
we would need to be on top of all of that and handle updates whenever
|
|
they are needed.
|
|
|
|
Another problem is the fact that Docker Machine, from its beginnings, was
|
|
focused on managing Linux based instances only. Despite that at some moment
|
|
Docker got official and native integration on Windows, Docker Machine never
|
|
followed this step. Nor its designed to make such integration easy.
|
|
|
|
There is also no support for MacOS. This one is obvious - Docker Machine is a
|
|
tool to maintain hosts for Docker Engine and there is no native Docker Engine
|
|
for MacOS. And by native we mean MacOS containers executed within MacOS
|
|
operating system. Docker for MacOS product is not a native support - it's just
|
|
a tooling and a virtualized Linux instance installed with it that makes it
|
|
easier to develop **Linux containers** on MacOS development instances.
|
|
|
|
This means that only one of three of our officially supported platforms -
|
|
Linux, Windows and MacOS - have a fully-featured support for CI/CD
|
|
auto-scaling. For Windows there is a possibility to use Kubernetes (which in
|
|
some cases have limitations) and maybe with a lot of effort we could bring
|
|
support for Windows into Docker Machine. But for MacOS, there is no
|
|
auto-scaling solution provided natively by GitLab Runner.
|
|
|
|
This is a huge limitation for our users and a frequently requested feature.
|
|
It's also a limitation for our SaaS runners offering. We've maintained to
|
|
create some sort of auto-scaling for our SaaS Windows and SaaS MacOS runners
|
|
hacking around Custom executor. But experiences from past three years show
|
|
that it's not the best way of doing this. And yet, after this time, Windows
|
|
and MacOS runners autoscaling lacks a lot of performance and feature support
|
|
that we have with our SaaS Linux runners.
|
|
|
|
To keep supporting our customers and the wider community and to improve our
|
|
SaaS runners maintenance we need to design a new mechanism for GitLab Runner
|
|
auto-scaling. It not only needs to support auto-scaling, but it also needs to
|
|
do that in the way to enable us to build on top of it to improve efficiency,
|
|
reliability and availability.
|
|
|
|
## Proposal
|
|
|
|
Currently, GitLab Runner auto-scaling can be configured in a few ways. Some
|
|
customers are successfully using an auto-scaled environment in Kubernetes. We
|
|
know that a custom and unofficial GitLab Runner version has been built to make
|
|
auto-scaling on Kubernetes more reliable. We recognize the importance of having
|
|
a really good Kubernetes solution for running multiple jobs in parallel, but
|
|
refinements in this area are out of scope for this architectural initiative.
|
|
|
|
We want to focus on resolving problems with Docker Machine and replacing this
|
|
mechanism with a reliable and flexible mechanism. We might be unable to build a
|
|
drop-in replacement for Docker Machine, as there are presumably many reasons
|
|
why it has been deprecated. It is very difficult to maintain compatibility with
|
|
so many cloud providers, and it seems that Docker Machine has been deprecated
|
|
in favor of Docker Desktop, which is not a viable replacement for us.
|
|
[This issue](https://github.com/docker/roadmap/issues/245) contains a discussion
|
|
about how people are using Docker Machine right now, and it seems that GitLab
|
|
CI is one of the most frequent reasons for people to keep using Docker Machine.
|
|
|
|
There is also an opportunity in being able to optionally run multiple jobs in a
|
|
single, larger virtual machine. We can't do that today, but we know that this
|
|
can potentially significantly improve efficiency. We might want to build a new
|
|
architecture that makes it easier and allows us to test how efficient it is
|
|
with PoCs. Running multiple jobs on a single machine can also make it possible
|
|
to reuse what we call a "sticky context" - a space for build artifacts / user
|
|
data that can be shared between job runs.
|
|
|
|
### 💡 Design a simple abstraction that users will be able to build on top of
|
|
|
|
Because there is no viable replacement and we might be unable to support all
|
|
cloud providers that Docker Machine used to support, the key design requirement
|
|
is to make it really simple and easy for the wider community to write a custom
|
|
GitLab plugin for whatever cloud provider they might be using. We
|
|
want to design a simple abstraction that users will be able to build on top, as
|
|
will we to support existing workflows on GitLab.com.
|
|
|
|
The designed mechanism should abstract what Docker Machine executor has been doing:
|
|
providing a way to create an external Docker environment, waiting to execute
|
|
jobs by provisioning this environment and returning credentials required to
|
|
perform these operations.
|
|
|
|
The new plugin system should be available for all major platforms: Linux,
|
|
Windows, MacOS.
|
|
|
|
### 💡 Migrate existing Docker Machine solution to a plugin
|
|
|
|
Once we design and implement the new abstraction, we should be able to migrate
|
|
existing Docker Machine mechanisms to a plugin. This will make it possible for
|
|
users and customers to immediately start using the new architecture, but still
|
|
keep their existing workflows and configuration for Docker Machine. This will
|
|
give everyone time to migrate to the new architecture before we drop support
|
|
for the legacy auto-scaling entirely.
|
|
|
|
### 💡 Build plugins for AWS, Google Cloud Platform and Azure
|
|
|
|
Although we might be unable to add support for all the cloud providers that
|
|
Docker Machine used to support, it seems to be important to provide
|
|
GitLab-maintained plugins for the major cloud providers like AWS, Google Cloud
|
|
Platform and Azure.
|
|
|
|
We should build them, presumably in separate repositories, in a way that they
|
|
are easy to contribute to, fork, modify for certain needs the wider community
|
|
team members might have. It should be also easy to install a new plugin without
|
|
the need of rebuilding GitLab Runner whenever it happens.
|
|
|
|
### 💡 Write a solid documentation about how to build your own plugin
|
|
|
|
It is important to show users how to build a plugin, so that they
|
|
can implement support for their own cloud infrastructure.
|
|
|
|
Building new plugins should be simple and supported with great
|
|
documentation. We want to design the plugin system in a way that the entry barrier
|
|
for contributing new plugins is very low.
|
|
|
|
### 💡 Build a PoC to run multiple builds on a single machine
|
|
|
|
We want to better understand what kind of efficiency can running multiple jobs
|
|
on a single machine bring. It is difficult to predict that, so ideally we
|
|
should build a PoC that will help us to better understand what we can expect
|
|
from this.
|
|
|
|
To run this experiment we most likely we will need to build an experimental
|
|
plugin, that not only allows us to schedule running multiple builds on a single
|
|
machine, but also has a set of comprehensive metrics built into it, to make it
|
|
easier to understand how it performs.
|
|
|
|
## Details
|
|
|
|
How the abstraction for the custom provider will look exactly is something that
|
|
we will need to prototype, PoC and decide in a data-informed way. There are a
|
|
few proposals that we should describe in detail, develop requirements for, PoC
|
|
and score. We will choose the solution that seems to support our goals the
|
|
most.
|
|
|
|
In order to describe the proposals we first need to better explain what part of
|
|
the GitLab Runner needs to be abstracted away. To make this easier to grasp
|
|
these concepts, let's take a look at the current auto-scaling architecture and
|
|
sequence diagram.
|
|
|
|
|
|
|
|
On the diagrams above we see that currently a GitLab Runner Manager runs on a
|
|
machine that has access to a cloud provider's API. It is using Docker Machine
|
|
to provision new Virtual Machines with Docker Engine installed and it
|
|
configures the Docker daemon there to allow external authenticated requests. It
|
|
stores credentials to such ephemeral Docker environments on disk. Once a
|
|
machine has been provisioned and made available for GitLab Runner Manager to
|
|
run builds, it is using one of the existing executors to run a user-provided
|
|
script. In auto-scaling, this is typically done using the Docker executor.
|
|
|
|
### Separation of concerns
|
|
|
|
There are several concerns represented in the current architecture. They are
|
|
coupled in the current implementation so we will break them out here to consider
|
|
them each separately.
|
|
|
|
- **Virtual Machine (VM) shape**. The underlying provider of a VM requires configuration to
|
|
know what kind of machine to create. E.g. Cores, memory, failure domain,
|
|
etc... This information is very provider specific.
|
|
- **VM lifecycle management**. Multiple machines will be created and a
|
|
system must keep track of which machines belong to this executor. Typically
|
|
a cloud provider will have a way to manage a set of homogenous machines.
|
|
E.g. GCE Instance Group. The basic operations are increase, decrease and
|
|
usually delete a specific machine.
|
|
- **VM autoscaling**. In addition to low-level lifecycle management,
|
|
job-aware capacity decisions must be made to the set of machines to provide
|
|
capacity when it is needed but not maintain excess capacity for cost reasons.
|
|
- **Job to VM mapping (routing)**. Currently the system assigns only one job to a
|
|
given a machine. A machine may be reused based on the specific executor
|
|
configuration.
|
|
- **In-VM job execution**. Within each VM a job must be driven through
|
|
various pre-defined stages and results and trace information returned
|
|
to the Runner system. These details are highly dependent on the VM
|
|
architecture and operating system as well as Executor type.
|
|
|
|
The current architecture has several points of coupling between concerns.
|
|
Coupling reduces opportunities for abstraction (e.g. community supported
|
|
plugins) and increases complexity, making the code harder to understand,
|
|
test, maintain and extend.
|
|
|
|
A primary design decision will be which concerns to externalize to the plugin
|
|
and which should remain with the runner system. The current implementation
|
|
has several abstractions internally which could be used as cut points for a
|
|
new abstraction.
|
|
|
|
For example the [`Build`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/267f40d871cd260dd063f7fbd36a921fedc62241/common/build.go#L125)
|
|
type uses the [`GetExecutorProvider`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/267f40d871cd260dd063f7fbd36a921fedc62241/common/executor.go#L171)
|
|
function to get an executor provider based on a dispatching executor string.
|
|
Various executor types register with the system by being imported and calling
|
|
[`RegisterExecutorProvider`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/267f40d871cd260dd063f7fbd36a921fedc62241/common/executor.go#L154)
|
|
during initialization. Here the abstractions are the [`ExecutorProvider`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/267f40d871cd260dd063f7fbd36a921fedc62241/common/executor.go#L80)
|
|
and [`Executor`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/267f40d871cd260dd063f7fbd36a921fedc62241/common/executor.go#L59)
|
|
interfaces.
|
|
|
|
Within the `docker+autoscaling` executor the [`machineExecutor`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/267f40d871cd260dd063f7fbd36a921fedc62241/executors/docker/machine/machine.go#L19)
|
|
type has a [`Machine`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/267f40d871cd260dd063f7fbd36a921fedc62241/helpers/docker/machine.go#L7)
|
|
interface which it uses to aquire a VM during the common [`Prepare`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/267f40d871cd260dd063f7fbd36a921fedc62241/executors/docker/machine/machine.go#L71)
|
|
phase. This abstraction primarily creates, accesses and deletes VMs.
|
|
|
|
There is no current abstraction for the VM autoscaling logic. It is tightly
|
|
coupled with the VM lifecycle and job routing logic. Creating idle capacity
|
|
happens as a side-effect of calling [`Acquire`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/267f40d871cd260dd063f7fbd36a921fedc62241/executors/docker/machine/provider.go#L449) on the `machineProvider` while binding a job to a VM.
|
|
|
|
There is also no current abstraction for in-VM job execution. VM-specific
|
|
commands are generated by the Runner Manager using the [`GenerateShellScript`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/267f40d871cd260dd063f7fbd36a921fedc62241/common/build.go#L336)
|
|
function and [injected](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/267f40d871cd260dd063f7fbd36a921fedc62241/common/build.go#L373)
|
|
into the VM as the manager drives the job execution stages.
|
|
|
|
### Design principles
|
|
|
|
Our goal is to design a GitLab Runner plugin system interface that is flexible
|
|
and simple for the wider community to consume. As we cannot build plugins for
|
|
all cloud platforms, we want to ensure a low entry barrier for anyone who needs
|
|
to develop a plugin. We want to allow everyone to contribute.
|
|
|
|
To achieve this goal, we will follow a few critical design principles. These
|
|
principles will guide our development process for the new plugin system
|
|
abstraction.
|
|
|
|
#### General high-level principles
|
|
|
|
- Design the new auto-scaling architecture aiming for having more choices and
|
|
flexibility in the future, instead of imposing new constraints.
|
|
- Design the new auto-scaling architecture to experiment with running multiple
|
|
jobs in parallel, on a single machine.
|
|
- Design the new provisioning architecture to replace Docker Machine in a way
|
|
that the wider community can easily build on top of the new abstractions.
|
|
- New auto-scaling method should become a core component of GitLab Runner product so that
|
|
we can simplify maintenance, use the same tooling, test configuration and Go language
|
|
setup as we do in our other main products.
|
|
- It should support multiple job execution environments - not only Docker containers
|
|
on Linux operating system.
|
|
|
|
The best design would be to bring auto-scaling as a feature wrapped around
|
|
our current executors like Docker or Shell.
|
|
|
|
#### Principles for the new plugin system
|
|
|
|
- Make the entry barrier for writing a new plugin low.
|
|
- Developing a new plugin should be simple and require only basic knowledge of
|
|
a programming language and a cloud provider's API.
|
|
- Strive for a balance between the plugin system's simplicity and flexibility.
|
|
These are not mutually exclusive.
|
|
- Abstract away as many technical details as possible but do not hide them completely.
|
|
- Build an abstraction that serves our community well but allows us to ship it quickly.
|
|
- Invest in a flexible solution, avoid one-way-door decisions, foster iteration.
|
|
- When in doubts err on the side of making things more simple for the wider community.
|
|
- Limit coupling between concerns to make the system more simple and extensible.
|
|
- Concerns should live on one side of the plug or the other--not both, which
|
|
duplicates effort and increases coupling.
|
|
|
|
#### The most important technical details
|
|
|
|
- Favor gRPC communication between a plugin and GitLab Runner.
|
|
- Make it possible to version communication interface and support many versions.
|
|
- Make Go a primary language for writing plugins but accept other languages too.
|
|
- Autoscaling mechanism should be fully owned by GitLab.
|
|
|
|
Cloud provider autoscalers don't know which VM to delete when scaling down so
|
|
they make sub-optimal decisions. Rather than teaching all autoscalers about GitLab
|
|
jobs, we prefer to have one, GitLab-owned autoscaler (not in the plugin).
|
|
|
|
It will also ensure that we can shape the future of the mechanism and make decisions
|
|
that fit our needs and requirements.
|
|
|
|
## Plugin boundary proposals
|
|
|
|
The following are proposals for where to draw the plugin boundary. We will evaluate
|
|
these proposals and others by the design principles and technical constraints
|
|
listed above.
|
|
|
|
### Custom provider
|
|
|
|
In order to reduce the scope of work, we only want to introduce the new
|
|
abstraction layer in one place.
|
|
|
|
A few years ago we introduced the [Custom Executor](https://docs.gitlab.com/runner/executors/custom.html)
|
|
feature in GitLab Runner. It allows users to design custom build execution
|
|
methods. The custom executor driver can be implemented in any way - from a
|
|
simple shell script to a dedicated binary - that is then used by a Runner
|
|
through os/exec system calls.
|
|
|
|
Thanks to the custom executor abstraction there is no more need to implement
|
|
new executors internally in Runner. Users who have specific needs can implement
|
|
their own drivers and don't need to wait for us to make their work part of the
|
|
"official" GitLab Runner. As each driver is a separate project, it also makes
|
|
it easier to create communities around them, where interested people can
|
|
collaborate together on improvements and bug fixes.
|
|
|
|
We want to design the new Custom Provider to replicate the success of the
|
|
Custom Executor. It will make it easier for users to build their own ways to
|
|
provide a context and an environment in which a build will be executed by one
|
|
of the Custom Executors.
|
|
|
|
There are multiple solutions to implementing a custom provider abstraction. We
|
|
can use raw Go plugins, Hashcorp's Go Plugin, HTTP interface or gRPC based
|
|
facade service. There are many solutions, and we want to choose the most
|
|
optimal one. In order to do that, we will describe the solutions in a separate
|
|
document, define requirements and score the solution accordingly. This will
|
|
allow us to choose a solution that will work best for us and the wider
|
|
community.
|
|
|
|
This proposal places VM lifecycle and autoscaling concerns as well as job to
|
|
VM mapping (routing) into the plugin. The build need only ask for a VM and
|
|
it will get one with all aspects of lifecycle and routing already accounted
|
|
for by the plugin.
|
|
|
|
Rationale: [Description of the Custom Executor Provider proposal](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28848#note_823321515)
|
|
|
|
### Fleeting VM provider
|
|
|
|
We can introduce a more simple version of the `Machine` abstraction in the
|
|
form of a "Fleeting" interface. Fleeting provides a low-level interface to
|
|
a homogenous VM group which allows increasing and decreasing the set size
|
|
as well as consuming a VM from within the set.
|
|
|
|
Plugins for cloud providers and other VM sources are implemented via the
|
|
Hashicorp go-plugin library. This is in practice gRPC over STDIN/STDOUT
|
|
but other wire protocols can be used also.
|
|
|
|
In order to make use of the new interface, the autoscaling logic is pulled
|
|
out of the Docker Executor and placed into a new Taskscaler library.
|
|
|
|
This places the concerns of VM lifecycle, VM shape and job routing within
|
|
the plugin. It also places the conern of VM autoscaling into a separate
|
|
component so it can be used by multiple Runner Executors (not just `docker+autoscaling`).
|
|
|
|
Rationale: [Description of the InstanceGroup / Fleeting proposal](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28848#note_823430883)
|
|
POC: [Merge request](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3315)
|
|
|
|
## Status
|
|
|
|
Status: RFC.
|
|
|
|
## Who
|
|
|
|
Proposal:
|
|
|
|
<!-- vale gitlab.Spelling = NO -->
|
|
|
|
| Role | Who |
|
|
|------------------------------|-------------------------------------------------|
|
|
| Authors | Grzegorz Bizon, Tomasz Maczukin, Joseph Burnett |
|
|
| Architecture Evolution Coach | Kamil Trzciński |
|
|
| Engineering Leader | Elliot Rushton, Cheryl Li |
|
|
| Product Manager | Darren Eastman, Jackie Porter |
|
|
| Domain Expert / Runner | Arran Walker |
|
|
|
|
DRIs:
|
|
|
|
| Role | Who |
|
|
|-------------|-----------------|
|
|
| Leadership | Elliot Rushton |
|
|
| Product | Darren Eastman |
|
|
| Engineering | Tomasz Maczukin |
|
|
|
|
Domain experts:
|
|
|
|
| Area | Who |
|
|
|------------------------|--------------|
|
|
| Domain Expert / Runner | Arran Walker |
|
|
|
|
<!-- vale gitlab.Spelling = YES -->
|