419 lines
16 KiB
Markdown
419 lines
16 KiB
Markdown
---
|
|
stage: Verify
|
|
group: Pipeline Authoring
|
|
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
|
|
type: reference
|
|
---
|
|
|
|
# Multi-project pipelines **(FREE)**
|
|
|
|
> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8.
|
|
|
|
You can set up [GitLab CI/CD](../index.md) across multiple projects, so that a pipeline
|
|
in one project can trigger a pipeline in another project. You can visualize the entire pipeline
|
|
in one place, including all cross-project interdependencies.
|
|
|
|
For example, you might deploy your web application from three different projects in GitLab.
|
|
Each project has its own build, test, and deploy process. With multi-project pipelines you can
|
|
visualize the entire pipeline, including all build and test stages for all three projects.
|
|
|
|
<i class="fa fa-youtube-play youtube" aria-hidden="true"></i>
|
|
For an overview, see the [Multi-project pipelines demo](https://www.youtube.com/watch?v=g_PIwBM1J84).
|
|
|
|
Multi-project pipelines are also useful for larger products that require cross-project interdependencies, like those
|
|
with a [microservices architecture](https://about.gitlab.com/blog/2016/08/16/trends-in-version-control-land-microservices/).
|
|
Learn more in the [Cross-project Pipeline Triggering and Visualization demo](https://about.gitlab.com/learn/)
|
|
at GitLab@learn, in the Continuous Integration section.
|
|
|
|
If you trigger a pipeline in a downstream private project, on the upstream project's pipelines page,
|
|
you can view:
|
|
|
|
- The name of the project.
|
|
- The status of the pipeline.
|
|
|
|
If you have a public project that can trigger downstream pipelines in a private project,
|
|
make sure there are no confidentiality problems.
|
|
|
|
## Create multi-project pipelines
|
|
|
|
To create multi-project pipelines, you can:
|
|
|
|
- [Define them in your `.gitlab-ci.yml` file](#define-multi-project-pipelines-in-your-gitlab-ciyml-file).
|
|
- [Use the API](#create-multi-project-pipelines-by-using-the-api).
|
|
|
|
### Define multi-project pipelines in your `.gitlab-ci.yml` file
|
|
|
|
> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8.
|
|
|
|
When you use the [`trigger`](../yaml/index.md#trigger) keyword to create a multi-project
|
|
pipeline in your `.gitlab-ci.yml` file, you create what is called a *trigger job*. For example:
|
|
|
|
```yaml
|
|
rspec:
|
|
stage: test
|
|
script: bundle exec rspec
|
|
|
|
staging:
|
|
variables:
|
|
ENVIRONMENT: staging
|
|
stage: deploy
|
|
trigger: my/deployment
|
|
```
|
|
|
|
In this example, after the `rspec` job succeeds in the `test` stage,
|
|
the `staging` trigger job starts. The initial status of this
|
|
job is `pending`.
|
|
|
|
GitLab then creates a downstream pipeline in the
|
|
`my/deployment` project and, as soon as the pipeline is created, the
|
|
`staging` job succeeds. The full path to the project is `my/deployment`.
|
|
|
|
You can view the status for the pipeline, or you can display
|
|
[the downstream pipeline's status instead](#mirror-status-of-a-triggered-pipeline-in-the-trigger-job).
|
|
|
|
The user that creates the upstream pipeline must be able to create pipelines in the
|
|
downstream project (`my/deployment`) too. If the downstream project is not found,
|
|
or the user does not have [permission](../../user/permissions.md) to create a pipeline there,
|
|
the `staging` job is marked as _failed_.
|
|
|
|
#### Trigger job configuration limitations
|
|
|
|
Trigger jobs can use only a limited set of the GitLab CI/CD [configuration keywords](../yaml/index.md).
|
|
The keywords available for use in trigger jobs are:
|
|
|
|
- [`trigger`](../yaml/index.md#trigger)
|
|
- [`stage`](../yaml/index.md#stage)
|
|
- [`allow_failure`](../yaml/index.md#allow_failure)
|
|
- [`rules`](../yaml/index.md#rules)
|
|
- [`only` and `except`](../yaml/index.md#only--except)
|
|
- [`when`](../yaml/index.md#when) (only with a value of `on_success`, `on_failure`, or `always`)
|
|
- [`extends`](../yaml/index.md#extends)
|
|
- [`needs`](../yaml/index.md#needs), but not [`needs:project`](../yaml/index.md#needsproject)
|
|
|
|
Trigger jobs cannot use [job-level persisted variables](../variables/where_variables_can_be_used.md#persisted-variables).
|
|
|
|
#### Specify a downstream pipeline branch
|
|
|
|
You can specify a branch name for the downstream pipeline to use.
|
|
GitLab uses the commit on the head of the branch to
|
|
create the downstream pipeline.
|
|
|
|
```yaml
|
|
rspec:
|
|
stage: test
|
|
script: bundle exec rspec
|
|
|
|
staging:
|
|
stage: deploy
|
|
trigger:
|
|
project: my/deployment
|
|
branch: stable-11-2
|
|
```
|
|
|
|
Use:
|
|
|
|
- The `project` keyword to specify the full path to a downstream project.
|
|
In [GitLab 15.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/367660), variable expansion is
|
|
supported.
|
|
- The `branch` keyword to specify the name of a branch in the project specified by `project`.
|
|
In [GitLab 12.4 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/10126), variable expansion is
|
|
supported.
|
|
|
|
Pipelines triggered on a protected branch in a downstream project use the [role](../../user/permissions.md)
|
|
of the user that ran the trigger job in the upstream project. If the user does not
|
|
have permission to run CI/CD pipelines against the protected branch, the pipeline fails. See
|
|
[pipeline security for protected branches](index.md#pipeline-security-on-protected-branches).
|
|
|
|
#### Pass CI/CD variables to a downstream pipeline by using the `variables` keyword
|
|
|
|
Sometimes you might want to pass CI/CD variables to a downstream pipeline.
|
|
You can do that by using the `variables` keyword, just like you would for any other job.
|
|
|
|
```yaml
|
|
rspec:
|
|
stage: test
|
|
script: bundle exec rspec
|
|
|
|
staging:
|
|
variables:
|
|
ENVIRONMENT: staging
|
|
stage: deploy
|
|
trigger: my/deployment
|
|
```
|
|
|
|
The `ENVIRONMENT` variable is passed to every job defined in a downstream
|
|
pipeline. It is available as a variable when GitLab Runner picks a job.
|
|
|
|
In the following configuration, the `MY_VARIABLE` variable is passed to the downstream pipeline
|
|
that is created when the `trigger-downstream` job is queued. This is because `trigger-downstream`
|
|
job inherits variables declared in global variables blocks, and then we pass these variables to a downstream pipeline.
|
|
|
|
```yaml
|
|
variables:
|
|
MY_VARIABLE: my-value
|
|
|
|
trigger-downstream:
|
|
variables:
|
|
ENVIRONMENT: something
|
|
trigger: my/project
|
|
```
|
|
|
|
You can stop global variables from reaching the downstream pipeline by using the [`inherit:variables` keyword](../yaml/index.md#inheritvariables).
|
|
In this example, the `MY_GLOBAL_VAR` variable is not available in the triggered pipeline:
|
|
|
|
```yaml
|
|
variables:
|
|
MY_GLOBAL_VAR: value
|
|
|
|
trigger-downstream:
|
|
inherit:
|
|
variables: false
|
|
variables:
|
|
MY_LOCAL_VAR: value
|
|
trigger: my/project
|
|
```
|
|
|
|
You might want to pass some information about the upstream pipeline using, for
|
|
example, predefined variables. In order to do that, you can use interpolation
|
|
to pass any variable. For example:
|
|
|
|
```yaml
|
|
downstream-job:
|
|
variables:
|
|
UPSTREAM_BRANCH: $CI_COMMIT_REF_NAME
|
|
trigger: my/project
|
|
```
|
|
|
|
In this scenario, the `UPSTREAM_BRANCH` variable with the value of the upstream pipeline's
|
|
`$CI_COMMIT_REF_NAME` is passed to `downstream-job`. It is available in the
|
|
context of all downstream builds.
|
|
|
|
You cannot use this method to forward [job-level persisted variables](../variables/where_variables_can_be_used.md#persisted-variables)
|
|
to a downstream pipeline, as they are not available in trigger jobs.
|
|
|
|
Upstream pipelines take precedence over downstream ones. If there are two
|
|
variables with the same name defined in both upstream and downstream projects,
|
|
the ones defined in the upstream project take precedence.
|
|
|
|
#### Pass CI/CD variables to a downstream pipeline by using variable inheritance **(PREMIUM)**
|
|
|
|
You can pass variables to a downstream pipeline with [`dotenv` variable inheritance](../variables/index.md#pass-an-environment-variable-to-another-job) and [`needs:project`](../yaml/index.md#needsproject).
|
|
|
|
In the upstream pipeline:
|
|
|
|
1. Save the variables in a `.env` file.
|
|
1. Save the `.env` file as a `dotenv` report.
|
|
1. Trigger the downstream pipeline.
|
|
|
|
```yaml
|
|
build_vars:
|
|
stage: build
|
|
script:
|
|
- echo "BUILD_VERSION=hello" >> build.env
|
|
artifacts:
|
|
reports:
|
|
dotenv: build.env
|
|
|
|
deploy:
|
|
stage: deploy
|
|
trigger: my/downstream_project
|
|
```
|
|
|
|
1. Set the `test` job in the downstream pipeline to inherit the variables from the `build_vars`
|
|
job in the upstream project with `needs`. The `test` job inherits the variables in the
|
|
`dotenv` report and it can access `BUILD_VERSION` in the script:
|
|
|
|
```yaml
|
|
test:
|
|
stage: test
|
|
script:
|
|
- echo $BUILD_VERSION
|
|
needs:
|
|
- project: my/upstream_project
|
|
job: build_vars
|
|
ref: master
|
|
artifacts: true
|
|
```
|
|
|
|
#### Pass artifacts to a downstream pipeline
|
|
|
|
You can pass artifacts to a downstream pipeline by using [`needs:project`](../yaml/index.md#needsproject).
|
|
|
|
1. In a job in the upstream pipeline, save the artifacts using the [`artifacts`](../yaml/index.md#artifacts) keyword.
|
|
1. Trigger the downstream pipeline with a trigger job:
|
|
|
|
```yaml
|
|
build_artifacts:
|
|
stage: build
|
|
script:
|
|
- echo "This is a test artifact!" >> artifact.txt
|
|
artifacts:
|
|
paths:
|
|
- artifact.txt
|
|
|
|
deploy:
|
|
stage: deploy
|
|
trigger: my/downstream_project
|
|
```
|
|
|
|
1. In a job in the downstream pipeline, fetch the artifacts from the upstream pipeline
|
|
by using `needs:project`. Set `job` to the job in the upstream pipeline to fetch artifacts from,
|
|
`ref` to the branch, and `artifacts: true`.
|
|
|
|
```yaml
|
|
test:
|
|
stage: test
|
|
script:
|
|
- cat artifact.txt
|
|
needs:
|
|
- project: my/upstream_project
|
|
job: build_artifacts
|
|
ref: main
|
|
artifacts: true
|
|
```
|
|
|
|
#### Pass artifacts to a downstream pipeline from a Merge Request pipeline
|
|
|
|
When you use `needs:project` to [pass artifacts to a downstream pipeline](#pass-artifacts-to-a-downstream-pipeline),
|
|
the `ref` value is usually a branch name, like `main` or `development`.
|
|
|
|
For merge request pipelines, the `ref` value is in the form of `refs/merge-requests/<id>/head`,
|
|
where `id` is the merge request ID. You can retrieve this ref with the [`CI_MERGE_REQUEST_REF_PATH`](../variables/predefined_variables.md#predefined-variables-for-merge-request-pipelines)
|
|
CI/CD variable. Do not use a branch name as the `ref` with merge request pipelines,
|
|
because the downstream pipeline attempts to fetch artifacts from the latest branch pipeline.
|
|
|
|
To fetch the artifacts from the upstream `merge request` pipeline instead of the `branch` pipeline,
|
|
pass this variable to the downstream pipeline using variable inheritance:
|
|
|
|
1. In a job in the upstream pipeline, save the artifacts using the [`artifacts`](../yaml/index.md#artifacts) keyword.
|
|
1. In the job that triggers the downstream pipeline, pass the `$CI_MERGE_REQUEST_REF_PATH` variable by using
|
|
[variable inheritance](#pass-cicd-variables-to-a-downstream-pipeline-by-using-the-variables-keyword):
|
|
|
|
```yaml
|
|
build_artifacts:
|
|
stage: build
|
|
script:
|
|
- echo "This is a test artifact!" >> artifact.txt
|
|
artifacts:
|
|
paths:
|
|
- artifact.txt
|
|
|
|
upstream_job:
|
|
variables:
|
|
UPSTREAM_REF: $CI_MERGE_REQUEST_REF_PATH
|
|
trigger:
|
|
project: my/downstream_project
|
|
branch: my-branch
|
|
```
|
|
|
|
1. In a job in the downstream pipeline, fetch the artifacts from the upstream pipeline
|
|
by using `needs:project`. Set the `ref` to the `UPSTREAM_REF` variable, and `job`
|
|
to the job in the upstream pipeline to fetch artifacts from:
|
|
|
|
```yaml
|
|
test:
|
|
stage: test
|
|
script:
|
|
- cat artifact.txt
|
|
needs:
|
|
- project: my/upstream_project
|
|
job: build_artifacts
|
|
ref: UPSTREAM_REF
|
|
artifacts: true
|
|
```
|
|
|
|
This method works for fetching artifacts from a regular merge request parent pipeline,
|
|
but fetching artifacts from [merge results](merged_results_pipelines.md) pipelines is not supported.
|
|
|
|
#### Use `rules` or `only`/`except` with multi-project pipelines
|
|
|
|
You can use CI/CD variables or the [`rules`](../yaml/index.md#rulesif) keyword to
|
|
[control job behavior](../jobs/job_control.md) for multi-project pipelines. When a
|
|
downstream pipeline is triggered with the [`trigger`](../yaml/index.md#trigger) keyword,
|
|
the value of the [`$CI_PIPELINE_SOURCE` predefined variable](../variables/predefined_variables.md)
|
|
is `pipeline` for all its jobs.
|
|
|
|
If you use [`only/except`](../yaml/index.md#only--except) to control job behavior, use the
|
|
[`pipelines`](../yaml/index.md#onlyrefs--exceptrefs) keyword.
|
|
|
|
#### Mirror status of a triggered pipeline in the trigger job
|
|
|
|
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11238) in GitLab Premium 12.3.
|
|
> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8.
|
|
|
|
You can mirror the pipeline status from the triggered pipeline to the source trigger job
|
|
by using [`strategy: depend`](../yaml/index.md#triggerstrategy). For example:
|
|
|
|
```yaml
|
|
trigger_job:
|
|
trigger:
|
|
project: my/project
|
|
strategy: depend
|
|
```
|
|
|
|
### Create multi-project pipelines by using the API
|
|
|
|
> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/31573) to GitLab Free in 12.4.
|
|
|
|
When you use the [`CI_JOB_TOKEN` to trigger pipelines](../jobs/ci_job_token.md),
|
|
GitLab recognizes the source of the job token. The pipelines become related,
|
|
so you can visualize their relationships on pipeline graphs.
|
|
|
|
These relationships are displayed in the pipeline graph by showing inbound and
|
|
outbound connections for upstream and downstream pipeline dependencies.
|
|
|
|
When using:
|
|
|
|
- CI/CD variables or [`rules`](../yaml/index.md#rulesif) to control job behavior, the value of
|
|
the [`$CI_PIPELINE_SOURCE` predefined variable](../variables/predefined_variables.md) is
|
|
`pipeline` for multi-project pipeline triggered through the API with `CI_JOB_TOKEN`.
|
|
- [`only/except`](../yaml/index.md#only--except) to control job behavior, use the
|
|
`pipelines` keyword.
|
|
|
|
## Trigger a pipeline when an upstream project is rebuilt **(PREMIUM)**
|
|
|
|
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9045) in GitLab 12.8.
|
|
|
|
You can trigger a pipeline in your project whenever a pipeline finishes for a new
|
|
tag in a different project.
|
|
|
|
Prerequisites:
|
|
|
|
- The upstream project must be [public](../../user/public_access.md).
|
|
- The user must have the Developer role
|
|
in the upstream project.
|
|
|
|
To trigger the pipeline when the upstream project is rebuilt:
|
|
|
|
1. On the top bar, select **Menu > Projects** and find your project.
|
|
1. On the left sidebar, select **Settings > CI/CD**.
|
|
1. Expand **Pipeline subscriptions**.
|
|
1. Enter the project you want to subscribe to, in the format `<namespace>/<project>`.
|
|
For example, if the project is `https://gitlab.com/gitlab-org/gitlab`, use `gitlab-org/gitlab`.
|
|
1. Select **Subscribe**.
|
|
|
|
Any pipelines that complete successfully for new tags in the subscribed project
|
|
now trigger a pipeline on the current project's default branch. The maximum
|
|
number of upstream pipeline subscriptions is 2 by default, for both the upstream and
|
|
downstream projects. On self-managed instances, an administrator can change this
|
|
[limit](../../administration/instance_limits.md#number-of-cicd-subscriptions-to-a-project).
|
|
|
|
## Multi-project pipeline visualization **(PREMIUM)**
|
|
|
|
When your pipeline triggers a downstream pipeline, the downstream pipeline displays
|
|
to the right of the [pipeline graph](index.md#visualize-pipelines).
|
|
|
|
|
|
|
|
In [pipeline mini graphs](index.md#pipeline-mini-graphs), the downstream pipeline
|
|
displays to the right of the mini graph.
|
|
|
|
|
|
|
|
## Retry or cancel multi-project pipelines
|
|
|
|
If you have permission to trigger pipelines in the downstream project, you can
|
|
retry or cancel multi-project pipelines:
|
|
|
|
- [In the main graph view](index.md#downstream-pipelines).
|
|
- From the downstream pipeline's details page.
|