Pipelines for the GitLab project

Pipelines for https://gitlab.com/gitlab-org/gitlab and https://gitlab.com/gitlab-org/gitlab-foss (as well as the dev instance's mirrors) are configured in the usual .gitlab-ci.yml which itself includes files under .gitlab/ci/ for easier maintenance.

We're striving to dogfood GitLab CI/CD features and best-practices as much as possible.

Stages

The current stages are:

sync: This stage is used to synchronize changes from https://gitlab.com/gitlab-org/gitlab to https://gitlab.com/gitlab-org/gitlab-foss.
prepare: This stage includes jobs that prepare artifacts that are needed by jobs in subsequent stages.
test: This stage includes most of the tests, DB/migration jobs, and static analysis jobs.
post-test: This stage includes jobs that build reports or gather data from the test stage's jobs (e.g. coverage, Knapsack metadata etc.).
review-prepare: This stage includes a job that build the CNG images that are later used by the (Helm) Review App deployment (see Review Apps for details).
review: This stage includes jobs that deploy the GitLab and Docs Review Apps.
qa: This stage includes jobs that perform QA tasks against the Review App that is deployed in the previous stage.
post-qa: This stage includes jobs that build reports or gather data from the qa stage's jobs (e.g. Review App performance report).
notification: This stage includes jobs that sends notifications about pipeline status.
pages: This stage includes a job that deploys the various reports as GitLab Pages (e.g. https://gitlab-org.gitlab.io/gitlab/coverage-ruby/, https://gitlab-org.gitlab.io/gitlab/coverage-javascript/, https://gitlab-org.gitlab.io/gitlab/webpack-report/).

Default image

The default image is currently registry.gitlab.com/gitlab-org/gitlab-build-images:ruby-2.6.5-golang-1.12-git-2.24-lfs-2.9-chrome-73.0-node-12.x-yarn-1.21-postgresql-9.6-graphicsmagick-1.3.34.

It includes Ruby 2.6.5, Go 1.12, Git 2.24, Git LFS 2.9, Chrome 73, Node 12, Yarn 1.21, PostgreSQL 9.6, and Graphics Magick 1.3.34.

The images used in our pipelines are configured in the gitlab-org/gitlab-build-images project, which is push-mirrored to https://dev.gitlab.org/gitlab/gitlab-build-images for redundancy.

The current version of the build images can be found in the "Used by GitLab section".

Default variables

In addition to the predefined variables, each pipeline includes default variables defined in https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml.

Common job definitions

Most of the jobs extend from a few CI definitions defined in .gitlab/ci/global.gitlab-ci.yml that are scoped to a single configuration parameter.

Job definitions	Description
`.default-tags`	Ensures a job has the `gitlab-org` tag to ensure it's using our dedicated runners.
`.default-retry`	Allows a job to retry upon `unknown_failure`, `api_failure`, `runner_system_failure`, `job_execution_timeout`, or `stuck_or_timeout_failure`.
`.default-before_script`	Allows a job to use a default `before_script` definition suitable for Ruby/Rails tasks that may need a database running (e.g. tests).
`.default-cache`	Allows a job to use a default `cache` definition suitable for Ruby/Rails and frontend tasks.
`.use-pg9`	Allows a job to use the `postgres:9.6.17` and `redis:alpine` services.
`.use-pg10`	Allows a job to use the `postgres:10.12` and `redis:alpine` services.
`.use-pg11`	Allows a job to use the `postgres:11.6` and `redis:alpine` services.
`.use-pg9-ee`	Same as `.use-pg9` but also use the `docker.elastic.co/elasticsearch/elasticsearch:6.4.2` services.
`.use-pg10-ee`	Same as `.use-pg10` but also use the `docker.elastic.co/elasticsearch/elasticsearch:6.4.2` services.
`.use-pg11-ee`	Same as `.use-pg11` but also use the `docker.elastic.co/elasticsearch/elasticsearch:6.4.2` services.
`.as-if-foss`	Simulate the FOSS project by setting the `FOSS_ONLY='1'` environment variable.

`workflow:rules`

We're using the workflow:rules keyword to define default rules to determine whether or not a pipeline is created.

These rules are defined in https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml and are as follows:

If $FORCE_GITLAB_CI is set, create a pipeline.
For merge requests, create a pipeline.
For master branch, create a pipeline (this includes on schedules, pushes, merges, etc.).
For tags, create a pipeline.
If $GITLAB_INTERNAL isn't set, don't create a pipeline.
For stable, auto-deploy, and security branches, create a pipeline.
For any other cases (e.g. when pushing a branch with no MR for it), no pipeline is created.

`rules`, `if:` conditions and `changes:` patterns

We're using the rules keyword extensively.

All rules definitions are defined in https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rules.gitlab-ci.yml, then included in individual jobs via extends.

The rules definitions are composed of if: conditions and changes: patterns, which are also defined in https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rules.gitlab-ci.yml and included in rules definitions via YAML anchors

`if:` conditions

`if:` conditions	Description	Notes
`if-not-canonical-namespace`	Matches if the project isn't in the canonical (`gitlab-org/`) or security (`gitlab-org/security`) namespace.	Use to create a job for forks (by using `when: on_success\|manual`), or not create a job for forks (by using `when: never`).
`if-not-ee`	Matches if the project isn't EE (i.e. project name isn't `gitlab` or `gitlab-ee`).	Use to create a job only in the FOSS project (by using `when: on_success
`if-not-foss`	Matches if the project isn't FOSS (i.e. project name isn't `gitlab-foss`, `gitlab-ce`, or `gitlabhq`).	Use to create a job only in the EE project (by using `when: on_success
`if-default-refs`	Matches if the pipeline is for `master`, `/^[\d-]+-stable(-ee)?$/` (stable branches), `/^\d+-\d+-auto-deploy-\d+$/` (auto-deploy branches), `/^security\//` (security branches), merge requests, and tags.	Note that jobs won't be created for branches with this default configuration.
`if-master-refs`	Matches if the current branch is `master`.
`if-master-or-tag`	Matches if the pipeline is for the `master` branch or for a tag.
`if-merge-request`	Matches if the pipeline is for a merge request.
`if-nightly-master-schedule`	Matches if the pipeline is for a `master` scheduled pipeline with `$NIGHTLY` set.
`if-dot-com-gitlab-org-schedule`	Limits jobs creation to scheduled pipelines for the `gitlab-org` group on GitLab.com.
`if-dot-com-gitlab-org-master`	Limits jobs creation to the `master` branch for the `gitlab-org` group on GitLab.com.
`if-dot-com-gitlab-org-merge-request`	Limits jobs creation to merge requests for the `gitlab-org` group on GitLab.com.
`if-dot-com-gitlab-org-and-security-tag`	Limits job creation to tags for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com.
`if-dot-com-gitlab-org-and-security-merge-request`	Limit jobs creation to merge requests for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com.
`if-dot-com-ee-schedule`	Limits jobs to scheduled pipelines for the `gitlab-org/gitlab` project on GitLab.com.
`if-cache-credentials-schedule`	Limits jobs to scheduled pipelines with the `$CI_REPO_CACHE_CREDENTIALS` variable set.

`changes:` patterns

`changes:` patterns	Description
`yaml-patterns`	Only create job for YAML-related changes.
`docs-patterns`	Only create job for docs-related changes.
`backstage-patterns`	Only create job for backstage-related changes.
`code-patterns`	Only create job for code-related changes.
`qa-patterns`	Only create job for QA-related changes.
`code-backstage-patterns`	Combination of `code-patterns` and `backstage-patterns`.
`code-qa-patterns`	Combination of `code-patterns` and `qa-patterns`.
`code-backstage-qa-patterns`	Combination of `code-patterns`, `backstage-patterns`, and `qa-patterns`.

Directed acyclic graph

We're using the needs: keyword to execute jobs out of order for the following jobs:

graph RL;
  A[setup-test-env];
  B["gitlab:assets:compile pull-push-cache<br/>(canonical master only)"];
  C["gitlab:assets:compile pull-cache<br/>(canonical default refs only)"];
  D["cache gems<br/>(master and tags only)"];
  E[review-build-cng];
  F[build-qa-image];
  G[review-deploy];
  I["karma, jest"];
  I2["karma-as-if-foss, jest-as-if-foss<br/>(EE default refs only)"];
  J["compile-assets pull-push-cache<br/>(master only)"];
  J2["compile-assets pull-push-cache as-if-foss<br/>(EE master only)"];
  K[compile-assets pull-cache];
  K2["compile-assets pull-cache as-if-foss<br/>(EE default refs only)"];
  U[frontend-fixtures];
  U2["frontend-fixtures-as-if-foss<br/>(EE default refs only)"];
  V["webpack-dev-server, static-analysis"];
  M[coverage];
  O[coverage-frontend];
  N["pages (master only)"];
  Q[package-and-qa];
  S["RSpec<br/>(e.g. rspec unit pg9)"]
  T[retrieve-tests-metadata];
  QA["qa:internal, qa:selectors"];
  QA2["qa:internal-as-if-foss, qa:selectors-as-if-foss<br/>(EE default refs only)"];
  X["docs lint, code_quality, sast, dependency_scanning, danger-review"];

subgraph "`prepare` stage"
    A
    B
    C
    F
    K
    K2
    J
    J2
    T
    end

subgraph "`fixture` stage"
    U -.-> |needs and depends on| A;
    U -.-> |needs and depends on| K;
    U2 -.-> |needs and depends on| A;
    U2 -.-> |needs and depends on| K2;
    end

subgraph "`test` stage"
    D -.-> |needs| A;
    I -.-> |needs and depends on| U;
    I2 -.-> |needs and depends on| U2;
    L -.-> |needs and depends on| A;
    S -.-> |needs and depends on| A;
    S -.-> |needs and depends on| K;
    S -.-> |needs and depends on| T;
    L["db:*, gitlab:setup, graphql-docs-verify, downtime_check"] -.-> |needs| A;
    V -.-> |needs and depends on| K;
    X -.-> |needs| T;
    QA -.-> |needs| T;
    QA2 -.-> |needs| T;
    end

subgraph "`post-test` stage"
    M --> |happens after| S
    O --> |needs `jest`| I
    end

subgraph "`review-prepare` stage"
    E -.-> |needs| C;
    end

subgraph "`review` stage"
    G -.-> |needs| E
    end

subgraph "`qa` stage"
    Q -.-> |needs| C;
    Q -.-> |needs| F;
    QA1["review-qa-smoke, review-qa-all, review-performance, dast"] -.-> |needs| G;
    end

subgraph "`post-qa` stage"
  PQA1["parallel-spec-reports"] -.-> |depends on `review-qa-all`| QA1;
  end

subgraph "`pages` stage"
    N -.-> |depends on| C;
    N -.-> |depends on karma| I;
    N -.-> |depends on| M;
    N --> |happens after| PQA1
    end

Test jobs

Consult GitLab tests in the Continuous Integration (CI) context for more information.

Review app jobs

Consult the Review Apps dedicated page for more information.

Return to Development documentation

13 KiB Raw Blame History