debian-mirror-gitlab/doc/ci/pipelines/parent_child_pipelines.md

219 lines
8.1 KiB
Markdown
Raw Normal View History

2021-09-30 23:02:18 +05:30
---
stage: Verify
group: Pipeline Authoring
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
type: reference
---
# Parent-child pipelines **(FREE)**
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16094) in GitLab 12.7.
As pipelines grow more complex, a few related problems start to emerge:
- The staged structure, where all steps in a stage must be completed before the first
job in next stage begins, causes arbitrary waits, slowing things down.
- Configuration for the single global pipeline becomes very long and complicated,
making it hard to manage.
- Imports with [`include`](../yaml/index.md#include) increase the complexity of the configuration, and create the potential
for namespace collisions where jobs are unintentionally duplicated.
- Pipeline UX can become unwieldy with so many jobs and stages to work with.
Additionally, sometimes the behavior of a pipeline needs to be more dynamic. The ability
to choose to start sub-pipelines (or not) is a powerful ability, especially if the
YAML is dynamically generated.
2021-11-18 22:05:49 +05:30
![Parent pipeline graph expanded](img/parent_pipeline_graph_expanded_v14_3.png)
2021-09-30 23:02:18 +05:30
Similarly to [multi-project pipelines](multi_project_pipelines.md), a pipeline can trigger a
set of concurrently running child pipelines, but within the same project:
- Child pipelines still execute each of their jobs according to a stage sequence, but
would be free to continue forward through their stages without waiting for unrelated
jobs in the parent pipeline to finish.
2022-01-26 12:08:38 +05:30
- The configuration is split up into smaller child pipeline configurations. Each child pipeline contains only relevant steps which are
2021-09-30 23:02:18 +05:30
easier to understand. This reduces the cognitive load to understand the overall configuration.
- Imports are done at the child pipeline level, reducing the likelihood of collisions.
Child pipelines work well with other GitLab CI/CD features:
- Use [`rules: changes`](../yaml/index.md#ruleschanges) to trigger pipelines only when
certain files change. This is useful for monorepos, for example.
- Since the parent pipeline in `.gitlab-ci.yml` and the child pipeline run as normal
pipelines, they can have their own behaviors and sequencing in relation to triggers.
2022-01-26 12:08:38 +05:30
See the [`trigger`](../yaml/index.md#trigger) keyword documentation for full details on how to
2021-09-30 23:02:18 +05:30
include the child pipeline configuration.
<i class="fa fa-youtube-play youtube" aria-hidden="true"></i>
For an overview, see [Parent-Child Pipelines feature demo](https://youtu.be/n8KpBSqZNbk).
## Examples
The simplest case is [triggering a child pipeline](../yaml/index.md#trigger) using a
local YAML file to define the pipeline configuration. In this case, the parent pipeline
triggers the child pipeline, and continues without waiting:
```yaml
microservice_a:
trigger:
include: path/to/microservice_a.yml
```
2021-11-11 11:23:49 +05:30
You can include multiple files when defining a child pipeline. The child pipeline's
configuration is composed of all configuration files merged together:
2021-09-30 23:02:18 +05:30
```yaml
microservice_a:
trigger:
include:
- local: path/to/microservice_a.yml
- template: Security/SAST.gitlab-ci.yml
```
2021-11-18 22:05:49 +05:30
In [GitLab 13.5 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/205157),
2021-09-30 23:02:18 +05:30
you can use [`include:file`](../yaml/index.md#includefile) to trigger child pipelines
with a configuration file in a different project:
```yaml
microservice_a:
trigger:
include:
- project: 'my-group/my-pipeline-library'
2021-12-11 22:18:48 +05:30
ref: 'main'
file: '/path/to/child-pipeline.yml'
2021-09-30 23:02:18 +05:30
```
2022-01-26 12:08:38 +05:30
The maximum number of entries that are accepted for `trigger:include` is three.
2021-09-30 23:02:18 +05:30
Similar to [multi-project pipelines](multi_project_pipelines.md#mirror-status-of-a-triggered-pipeline-in-the-trigger-job),
we can set the parent pipeline to depend on the status of the child pipeline upon completion:
```yaml
microservice_a:
trigger:
include:
- local: path/to/microservice_a.yml
- template: Security/SAST.gitlab-ci.yml
strategy: depend
```
2021-12-11 22:18:48 +05:30
## Merge request child pipelines
2021-09-30 23:02:18 +05:30
2022-04-04 11:22:00 +05:30
To trigger a child pipeline as a [merge request pipeline](merge_request_pipelines.md) we need to:
2021-09-30 23:02:18 +05:30
- Set the trigger job to run on merge requests:
```yaml
# parent .gitlab-ci.yml
microservice_a:
trigger:
include: path/to/microservice_a.yml
rules:
- if: $CI_MERGE_REQUEST_ID
```
- Configure the child pipeline by either:
- Setting all jobs in the child pipeline to evaluate in the context of a merge request:
```yaml
# child path/to/microservice_a.yml
workflow:
rules:
- if: $CI_MERGE_REQUEST_ID
job1:
script: ...
job2:
script: ...
```
- Alternatively, setting the rule per job. For example, to create only `job1` in
the context of merge request pipelines:
```yaml
# child path/to/microservice_a.yml
job1:
script: ...
rules:
- if: $CI_MERGE_REQUEST_ID
job2:
script: ...
```
## Dynamic child pipelines
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35632) in GitLab 12.9.
Instead of running a child pipeline from a static YAML file, you can define a job that runs
2021-12-11 22:18:48 +05:30
your own script to generate a YAML file, which is then used to trigger a child pipeline.
2021-09-30 23:02:18 +05:30
This technique can be very powerful in generating pipelines targeting content that changed or to
build a matrix of targets and architectures.
<i class="fa fa-youtube-play youtube" aria-hidden="true"></i>
For an overview, see [Create child pipelines using dynamically generated configurations](https://youtu.be/nMdfus2JWHM).
We also have an example project using
[Dynamic Child Pipelines with Jsonnet](https://gitlab.com/gitlab-org/project-templates/jsonnet)
2021-10-27 15:23:28 +05:30
which shows how to use a data templating language to generate your `.gitlab-ci.yml` at runtime.
You could use a similar process for other templating languages like
2021-11-11 11:23:49 +05:30
[Dhall](https://dhall-lang.org/) or [ytt](https://get-ytt.io/).
2021-09-30 23:02:18 +05:30
The artifact path is parsed by GitLab, not the runner, so the path must match the
syntax for the OS running GitLab. If GitLab is running on Linux but using a Windows
runner for testing, the path separator for the trigger job would be `/`. Other CI/CD
configuration for jobs, like scripts, that use the Windows runner would use `\`.
In GitLab 12.9, the child pipeline could fail to be created in certain cases, causing the parent pipeline to fail.
2021-11-18 22:05:49 +05:30
This is [resolved](https://gitlab.com/gitlab-org/gitlab/-/issues/209070) in GitLab 12.10.
2021-09-30 23:02:18 +05:30
2021-12-11 22:18:48 +05:30
### Dynamic child pipeline example
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35632) in GitLab 12.9.
You can trigger a child pipeline from a [dynamically generated configuration file](../pipelines/parent_child_pipelines.md#dynamic-child-pipelines):
```yaml
generate-config:
stage: build
script: generate-ci-config > generated-config.yml
artifacts:
paths:
- generated-config.yml
child-pipeline:
stage: test
trigger:
include:
- artifact: generated-config.yml
job: generate-config
```
The `generated-config.yml` is extracted from the artifacts and used as the configuration
for triggering the child pipeline.
2021-09-30 23:02:18 +05:30
## Nested child pipelines
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29651) in GitLab 13.4.
> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/243747) in GitLab 13.5.
Parent and child pipelines were introduced with a maximum depth of one level of child
pipelines, which was later increased to two. A parent pipeline can trigger many child
pipelines, and these child pipelines can trigger their own child pipelines. It's not
possible to trigger another level of child pipelines.
<i class="fa fa-youtube-play youtube" aria-hidden="true"></i>
For an overview, see [Nested Dynamic Pipelines](https://youtu.be/C5j3ju9je2M).
## Pass CI/CD variables to a child pipeline
You can pass CI/CD variables to a downstream pipeline using the same methods as
multi-project pipelines:
- [By using the `variable` keyword](multi_project_pipelines.md#pass-cicd-variables-to-a-downstream-pipeline-by-using-the-variables-keyword).
- [By using variable inheritance](multi_project_pipelines.md#pass-cicd-variables-to-a-downstream-pipeline-by-using-variable-inheritance).