debian-mirror-gitlab/doc/ci/runners/saas/linux_saas_runner.md

---
stage: Verify
group: Runner
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---

# SaaS runners on Linux

When you run jobs on SaaS runners on Linux, the runners are on auto-scaled ephemeral virtual machine (VM) instances.
Each VM uses the Google Container-Optimized OS (COS) and the latest version of Docker Engine.
The default region for the VMs is `us-east1`.

## Machine types available for private projects (x86-64)

For the SaaS runners on Linux, GitLab offers a range of machine types for use in private projects.
For Free, Premium, and Ultimate plan customers, jobs on these instances consume the CI/CD minutes allocated to your namespace.

|                   | Small                     | Medium                    | Large                    |
|-------------------|---------------------------|---------------------------|--------------------------|
| Specs             | 1 vCPU, 3.75 GB RAM        | 2 vCPUs, 8 GB RAM          | 4 vCPUs, 16 GB RAM        |
| GitLab CI/CD tags | `saas-linux-small-amd64` | `saas-linux-medium-amd64` | `saas-linux-large-amd64` |
| Subscription      | Free, Premium, Ultimate   | Free, Premium, Ultimate   | Premium, Ultimate        |

The `small` machine type is the default. Your job runs on this machine type if you don't specify
a [tags:](../../yaml/index.md#tags) keyword in your `.gitlab-ci.yml` file.

CI/CD jobs that run on `medium` and `large` machine types consumes CI minutes at a different rate than CI/CD jobs on the `small` machine type.

Refer to the CI/CD minutes [cost factor](../../../ci/pipelines/cicd_minutes.md#cost-factor) for the cost factor applied to the machine type based on size.

## GPU-enabled SaaS runners on Linux **(PREMIUM SAAS)**

We offer GPU-enabled SaaS runners for heavy compute including ModelOps or HPC workloads. Available to Premium and Ultimate plan customers, jobs on these instances consume the CI/CD minutes allocated to your namespace.

|                   | Standard                   |
|-------------------|---------------------------|
| Specs             | 4 vCPU, 16 GB RAM, 1 Nvidia Tesla T4 GPU (or similar) |
| GitLab CI/CD tags | `saas-linux-medium-gpu-standard` |

## Example of how to tag a job

To use a machine type other than `small`, add a `tags:` keyword to your job.
For example:

```yaml
stages:
  - Prebuild
  - Build
  - Unit Test

job_001:
 stage: Prebuild
 script:
  - echo "this job runs on the default (small) instance"

job_002:
 tags: [ saas-linux-medium-amd64 ]
 stage: Build
 script:
  - echo "this job runs on the medium instance"


job_003:
 tags: [ saas-linux-large-amd64 ]
 stage: Unit Test
 script:
  - echo "this job runs on the large instance"

```

## SaaS runners for GitLab projects

The `gitlab-shared-runners-manager-X.gitlab.com` fleet of runners are dedicated for
GitLab projects and related community forks. These runners are backed by a Google Compute
`n1-standard-2` machine type and do not run untagged jobs. Unlike the machine types used
for private projects, each virtual machine is re-used up to 40 times.

## SaaS runners on Linux settings

Below are the settings for SaaS runners on Linux.

| Setting                                                                 | GitLab.com       | Default |
|-------------------------------------------------------------------------|------------------|---------|
| Executor                                                                | `docker+machine` | -       |
| Default Docker image                                                    | `ruby:3.1`       | -       |
| `privileged` (run [Docker in Docker](https://hub.docker.com/_/docker/)) | `true`           | `false` |

- **Cache**: These runners share a
  [distributed cache](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching)
  that's stored in a Google Cloud Storage (GCS) bucket. Cache contents not updated in
  the last 14 days are automatically removed, based on the
  [object lifecycle management policy](https://cloud.google.com/storage/docs/lifecycle).

- **Timeout settings**: Jobs handled by the SaaS Runners on Linux
  **time out after 3 hours**, regardless of the timeout configured in a
  project. For details, see issues [#4010](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/4010)
  and [#4070](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/4070).

NOTE:
SaaS runner instances are provisioned with a 25 GB storage volume. The underlying disk space of the storage volume
is shared by the operating system, the Docker image, and a copy of your cloned repository.
This means that the available free disk space that your jobs can use is **less than 25 GB**.

## Pre-clone script (deprecated)

WARNING:
This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/391896) in GitLab 15.9
and is planned for removal in 16.0. Use [`pre_get_sources_script`](../../../ci/yaml/index.md#hookspre_get_sources_script) instead. This change is a breaking change.
With SaaS runners on Linux, you can run commands in a CI/CD
job before the runner attempts to run `git init` and `git fetch` to
download a GitLab repository. The
[`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section)
can be used for:

- Seeding the build directory with repository data
- Sending a request to a server
- Downloading assets from a CDN
- Any other commands that must run before the `git init`

To use this feature, define a [CI/CD variable](../../../ci/variables/index.md) called
`CI_PRE_CLONE_SCRIPT` that contains a bash script.

NOTE:
The `CI_PRE_CLONE_SCRIPT` variable does not work on GitLab SaaS Windows or macOS runners.

### Pre-clone script example

This example was used in the `gitlab-org/gitlab` project until November 2021.
The project no longer uses this optimization because the
[pack-objects cache](../../../administration/gitaly/configure_gitaly.md#pack-objects-cache)
lets Gitaly serve the full CI/CD fetch traffic. See [Git fetch caching](../../../development/pipelines/performance.md#git-fetch-caching).

The `CI_PRE_CLONE_SCRIPT` was defined as a project CI/CD variable:

```shell
(
  echo "Downloading archived master..."
  wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master-shallow.tar.gz

  if [ ! -f /tmp/gitlab.tar.gz ]; then
      echo "Repository cache not available, cloning a new directory..."
      exit
  fi

  rm -rf $CI_PROJECT_DIR
  echo "Extracting tarball into $CI_PROJECT_DIR..."
  mkdir -p $CI_PROJECT_DIR
  cd $CI_PROJECT_DIR
  tar xzf /tmp/gitlab.tar.gz
  rm -f /tmp/gitlab.tar.gz
  chmod a+w $CI_PROJECT_DIR
)
```

The first step of the script downloads `gitlab-master.tar.gz` from Google Cloud Storage.
There was a [GitLab CI/CD job named `cache-repo`](https://gitlab.com/gitlab-org/gitlab/-/blob/5fb40526c8c8aaafc5f92eab36d5bbddaca3893d/.gitlab/ci/cache-repo.gitlab-ci.yml)
that was responsible for keeping that archive up-to-date. Every two hours on a scheduled pipeline,
it did the following:

1. Create a fresh clone of the `gitlab-org/gitlab` repository on GitLab.com.
1. Save the data as a `.tar.gz`.
1. Upload it into the Google Cloud Storage bucket.

When a job ran with this configuration, the output looked similar to:

```shell
$ eval "$CI_PRE_CLONE_SCRIPT"
Downloading archived master...
Extracting tarball into /builds/gitlab-org/gitlab...
Fetching changes...
Reinitialized existing Git repository in /builds/gitlab-org/gitlab/.git/
```

The `Reinitialized existing Git repository` message shows that
the pre-clone step worked. The runner runs `git init`, which
overwrites the Git configuration with the appropriate settings to fetch
from the GitLab repository.

`CI_REPO_CACHE_CREDENTIALS` must contain the Google Cloud service account
JSON for uploading to the `gitlab-ci-git-repo-cache` bucket.

This bucket should be located in the same continent as the
runner, or [you can incur network egress charges](https://cloud.google.com/storage/pricing).

## `config.toml`

The full contents of our `config.toml` are:

NOTE:
Settings that are not public are shown as `X`.

**Google Cloud Platform**

```toml
concurrent = X
check_interval = 1
metrics_server = "X"
sentry_dsn = "X"

[[runners]]
  name = "docker-auto-scale"
  request_concurrency = X
  url = "https://gitlab.com/"
  token = "SHARED_RUNNER_TOKEN"
  pre_clone_script = "eval \"$CI_PRE_CLONE_SCRIPT\""
  executor = "docker+machine"
  environment = [
    "DOCKER_DRIVER=overlay2",
    "DOCKER_TLS_CERTDIR="
  ]
  limit = X
  [runners.docker]
    image = "ruby:3.1"
    privileged = true
    volumes = [
      "/certs/client",
      "/dummy-sys-class-dmi-id:/sys/class/dmi/id:ro" # Make kaniko builds work on GCP.
    ]
  [runners.machine]
    IdleCount = 50
    IdleTime = 3600
    MaxBuilds = 1 # For security reasons we delete the VM after job has finished so it's not reused.
    MachineName = "srm-%s"
    MachineDriver = "google"
    MachineOptions = [
      "google-project=PROJECT",
      "google-disk-size=25",
      "google-machine-type=n1-standard-1",
      "google-username=core",
      "google-tags=gitlab-com,srm",
      "google-use-internal-ip",
      "google-zone=us-east1-d",
      "engine-opt=mtu=1460", # Set MTU for container interface, for more information check https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3214#note_82892928
      "google-machine-image=PROJECT/global/images/IMAGE",
      "engine-opt=ipv6", # This will create IPv6 interfaces in the containers.
      "engine-opt=fixed-cidr-v6=fc00::/7",
      "google-operation-backoff-initial-interval=2" # Custom flag from forked docker-machine, for more information check https://github.com/docker/machine/pull/4600
    ]
    [[runners.machine.autoscaling]]
      Periods = ["* * * * * sat,sun *"]
      Timezone = "UTC"
      IdleCount = 70
      IdleTime = 3600
    [[runners.machine.autoscaling]]
      Periods = ["* 30-59 3 * * * *", "* 0-30 4 * * * *"]
      Timezone = "UTC"
      IdleCount = 700
      IdleTime = 3600
  [runners.cache]
    Type = "gcs"
    Shared = true
    [runners.cache.gcs]
      CredentialsFile = "/path/to/file"
      BucketName = "bucket-name"
```