debian-mirror-gitlab/doc/development/pipelines/internals.md

stage	group	info
none	Engineering Productivity	To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments

CI configuration internals

Workflow rules

Pipelines for the GitLab project are created using the workflow:rules keyword feature of the GitLab CI/CD.

Pipelines are always created for the following scenarios:

• main branch, including on schedules, pushes, merges, and so on.
• Merge requests.
• Tags.
• Stable, auto-deploy, and security branches.

Pipeline creation is also affected by the following CI/CD variables:

• If $FORCE_GITLAB_CI is set, pipelines are created.
• If $GITLAB_INTERNAL is not set, pipelines are not created.

No pipeline is created in any other cases (for example, when pushing a branch with no MR for it).

The source of truth for these workflow rules is defined in .gitlab-ci.yml.

Default image

The default image is defined in .gitlab-ci.yml.

It includes Ruby, Go, Git, Git LFS, Chrome, Node, Yarn, PostgreSQL, and Graphics Magick.

The images used in our pipelines are configured in the gitlab-org/gitlab-build-images project, which is push-mirrored to gitlab/gitlab-build-images for redundancy.

The current version of the build images can be found in the "Used by GitLab section".

Default variables

In addition to the predefined CI/CD variables, each pipeline includes default variables defined in .gitlab-ci.yml.

Stages

The current stages are:

• sync: This stage is used to synchronize changes from gitlab-org/gitlab to gitlab-org/gitlab-foss.
• prepare: This stage includes jobs that prepare artifacts that are needed by jobs in subsequent stages.
• build-images: This stage includes jobs that prepare Docker images that are needed by jobs in subsequent stages or downstream pipelines.
• fixtures: This stage includes jobs that prepare fixtures needed by frontend tests.
• lint: This stage includes linting and static analysis jobs.
• test: This stage includes most of the tests, and DB/migration jobs.
• post-test: This stage includes jobs that build reports or gather data from the test stage's jobs (for example, coverage, Knapsack metadata, and so on).
• review: This stage includes jobs that build the CNG images, deploy them, and run end-to-end tests against Review Apps (see Review Apps for details). It also includes Docs Review App jobs.
• qa: This stage includes jobs that perform QA tasks against the Review App that is deployed in stage review.
• post-qa: This stage includes jobs that build reports or gather data from the qa stage's jobs (for example, Review App performance report).
• pages: This stage includes a job that deploys the various reports as GitLab Pages (for example, coverage-ruby, and webpack-report (found at https://gitlab-org.gitlab.io/gitlab/webpack-report/, but there is an issue with the deployment).
• notify: This stage includes jobs that notify various failures to Slack.

Dependency Proxy

Some of the jobs are using images from Docker Hub, where we also use ${GITLAB_DEPENDENCY_PROXY_ADDRESS} as a prefix to the image path, so that we pull images from our Dependency Proxy. By default, this variable is set from the value of ${GITLAB_DEPENDENCY_PROXY}.

${GITLAB_DEPENDENCY_PROXY} is a group CI/CD variable defined in gitlab-org as ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/. This means when we use an image defined as:

image: ${GITLAB_DEPENDENCY_PROXY_ADDRESS}alpine:edge

Projects in the gitlab-org group pull from the Dependency Proxy, while forks that reside on any other personal namespaces or groups fall back to Docker Hub unless ${GITLAB_DEPENDENCY_PROXY} is also defined there.

Work around for when a pipeline is started by a Project access token user

When a pipeline is started by a Project access token user (e.g. the release-tools approver bot user which automatically updates the Gitaly version used in the main project), the Dependency proxy isn't accessible and the job fails at the Preparing the "docker+machine" executor step. To work around that, we have a special workflow rule, that overrides the ${GITLAB_DEPENDENCY_PROXY_ADDRESS} variable so that Dependency proxy isn't used in that case:

- if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $GITLAB_USER_LOGIN =~ /project_\d+_bot\d*/'
  variables:
    GITLAB_DEPENDENCY_PROXY_ADDRESS: ""

NOTE: We don't directly override the ${GITLAB_DEPENDENCY_PROXY} variable because group-level variables have higher precedence over .gitlab-ci.yml variables.

Common job definitions

Most of the jobs extend from a few CI definitions defined in .gitlab/ci/global.gitlab-ci.yml that are scoped to a single configuration keyword.

Job definitions	Description
.default-retry	Allows a job to retry upon unknown_failure, api_failure, runner_system_failure, job_execution_timeout, or stuck_or_timeout_failure.
.default-before_script	Allows a job to use a default before_script definition suitable for Ruby/Rails tasks that may need a database running (for example, tests).
.setup-test-env-cache	Allows a job to use a default cache definition suitable for setting up test environment for subsequent Ruby/Rails tasks.
.rails-cache	Allows a job to use a default cache definition suitable for Ruby/Rails tasks.
.static-analysis-cache	Allows a job to use a default cache definition suitable for static analysis tasks.
.coverage-cache	Allows a job to use a default cache definition suitable for coverage tasks.
.qa-cache	Allows a job to use a default cache definition suitable for QA tasks.
.yarn-cache	Allows a job to use a default cache definition suitable for frontend jobs that do a yarn install.
.assets-compile-cache	Allows a job to use a default cache definition suitable for frontend jobs that compile assets.
.use-pg13	Allows a job to use the postgres 13 and redis services (see .gitlab/ci/global.gitlab-ci.yml for the specific versions of the services).
.use-pg13-ee	Same as .use-pg13 but also use an elasticsearch service (see .gitlab/ci/global.gitlab-ci.yml for the specific version of the service).
.use-pg14	Allows a job to use the postgres 14 and redis services (see .gitlab/ci/global.gitlab-ci.yml for the specific versions of the services).
.use-pg14-ee	Same as .use-pg14 but also use an elasticsearch service (see .gitlab/ci/global.gitlab-ci.yml for the specific version of the service).
.use-kaniko	Allows a job to use the kaniko tool to build Docker images.
.as-if-foss	Simulate the FOSS project by setting the FOSS_ONLY='1' CI/CD variable.
.use-docker-in-docker	Allows a job to use Docker in Docker.

rules, if: conditions and changes: patterns

We're using the rules keyword extensively.

All rules definitions are defined in rules.gitlab-ci.yml, then included in individual jobs via extends.

The rules definitions are composed of if: conditions and changes: patterns, which are also defined in rules.gitlab-ci.yml and included in rules definitions via YAML anchors

if: conditions

if: conditions	Description	Notes
if-not-canonical-namespace	Matches if the project isn't in the canonical (gitlab-org/) or security (gitlab-org/security) namespace.	Use to create a job for forks (by using when: on_success or when: manual), or not create a job for forks (by using when: never).
if-not-ee	Matches if the project isn't EE (that is, project name isn't gitlab or gitlab-ee).	Use to create a job only in the FOSS project (by using when: on_success or when: manual), or not create a job if the project is EE (by using when: never).
if-not-foss	Matches if the project isn't FOSS (that is, project name isn't gitlab-foss, gitlab-ce, or gitlabhq).	Use to create a job only in the EE project (by using when: on_success or when: manual), or not create a job if the project is FOSS (by using when: never).
if-default-refs	Matches if the pipeline is for master, main, /^[\d-]+-stable(-ee)?$/ (stable branches), /^\d+-\d+-auto-deploy-\d+$/ (auto-deploy branches), /^security\// (security branches), merge requests, and tags.	Note that jobs aren't created for branches with this default configuration.
if-master-refs	Matches if the current branch is master or main.
if-master-push	Matches if the current branch is master or main and pipeline source is push.
if-master-schedule-maintenance	Matches if the current branch is master or main and pipeline runs on a 2-hourly schedule.
if-master-schedule-nightly	Matches if the current branch is master or main and pipeline runs on a nightly schedule.
if-auto-deploy-branches	Matches if the current branch is an auto-deploy one.
if-master-or-tag	Matches if the pipeline is for the master or main branch or for a tag.
if-merge-request	Matches if the pipeline is for a merge request.
if-merge-request-title-as-if-foss	Matches if the pipeline is for a merge request and the MR has label ~"pipeline:run-as-if-foss"
if-merge-request-title-update-caches	Matches if the pipeline is for a merge request and the MR has label ~"pipeline:update-cache".
if-merge-request-title-run-all-rspec	Matches if the pipeline is for a merge request and the MR has label ~"pipeline:run-all-rspec".
if-security-merge-request	Matches if the pipeline is for a security merge request.
if-security-schedule	Matches if the pipeline is for a security scheduled pipeline.
if-nightly-master-schedule	Matches if the pipeline is for a master scheduled pipeline with $NIGHTLY set.
if-dot-com-gitlab-org-schedule	Limits jobs creation to scheduled pipelines for the gitlab-org group on GitLab.com.
if-dot-com-gitlab-org-master	Limits jobs creation to the master or main branch for the gitlab-org group on GitLab.com.
if-dot-com-gitlab-org-merge-request	Limits jobs creation to merge requests for the gitlab-org group on GitLab.com.
if-dot-com-gitlab-org-and-security-tag	Limits job creation to tags for the gitlab-org and gitlab-org/security groups on GitLab.com.
if-dot-com-gitlab-org-and-security-merge-request	Limit jobs creation to merge requests for the gitlab-org and gitlab-org/security groups on GitLab.com.
if-dot-com-gitlab-org-and-security-tag	Limit jobs creation to tags for the gitlab-org and gitlab-org/security groups on GitLab.com.
if-dot-com-ee-schedule	Limits jobs to scheduled pipelines for the gitlab-org/gitlab project on GitLab.com.

changes: patterns

changes: patterns	Description
ci-patterns	Only create job for CI configuration-related changes.
ci-build-images-patterns	Only create job for CI configuration-related changes related to the build-images stage.
ci-review-patterns	Only create job for CI configuration-related changes related to the review stage.
ci-qa-patterns	Only create job for CI configuration-related changes related to the qa stage.
yaml-lint-patterns	Only create job for YAML-related changes.
docs-patterns	Only create job for docs-related changes.
frontend-dependency-patterns	Only create job when frontend dependencies are updated (for example, package.json, and yarn.lock) changes.
frontend-patterns-for-as-if-foss	Only create job for frontend-related changes that have impact on FOSS.
backend-patterns	Only create job for backend-related changes.
db-patterns	Only create job for DB-related changes.
backstage-patterns	Only create job for backstage-related changes (that is, Danger, fixtures, RuboCop, specs).
code-patterns	Only create job for code-related changes.
qa-patterns	Only create job for QA-related changes.
code-backstage-patterns	Combination of code-patterns and backstage-patterns.
code-qa-patterns	Combination of code-patterns and qa-patterns.
code-backstage-qa-patterns	Combination of code-patterns, backstage-patterns, and qa-patterns.
static-analysis-patterns	Only create jobs for Static Analytics configuration-related changes.

Best Practices

When to use extends:, <<: *xyz (YAML anchors), or !reference

Reference

Key takeaways

• If you need to extend a hash, you should use extends
• If you need to extend an array, you'll need to use !reference, or YAML anchors as last resort
• For more complex cases (e.g. extend hash inside array, extend array inside hash, ...), you'll have to use !reference or YAML anchors

What can extends and YAML anchors do?

extends

• Deep merge for hashes
• NO merge for arrays. It overwrites (source)

YAML anchors

• NO deep merge for hashes, BUT it can be used to extend a hash (see the example below)
• NO merge for arrays, BUT it can be used to extend an array (see the example below)

A great example

This example shows how to extend complex YAML data structures with !reference and YAML anchors:

.strict-ee-only-rules:
  # `rules` is an array of hashes
  rules:
    - if: '$CI_PROJECT_NAME !~ /^gitlab(-ee)?$/ '
      when: never

# `if-security-merge-request` is a hash
.if-security-merge-request: &if-security-merge-request
  if: '$CI_PROJECT_NAMESPACE == "gitlab-org/security"'

# `code-qa-patterns` is an array
.code-qa-patterns: &code-qa-patterns
  - "{package.json,yarn.lock}"
  - ".browserslistrc"
  - "babel.config.js"
  - "jest.config.{base,integration,unit}.js"

.qa:rules:as-if-foss:
  rules:
    # We extend the `rules` array with an array of hashes directly
    - !reference [".strict-ee-only-rules", rules]
    # We extend a single array entry with a hash
    - <<: *if-security-merge-request
      # `changes` is an array, so we pass it an entire array
      changes: *code-qa-patterns

qa:selectors-as-if-foss:
  # We include the rules from .qa:rules:as-if-foss in this job
  extends:
    - .qa:rules:as-if-foss