446 lines
22 KiB
Markdown
446 lines
22 KiB
Markdown
# Auto DevOps
|
||
|
||
> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/37115) in GitLab 10.0.
|
||
> - Generally available on GitLab 11.0.
|
||
|
||
Auto DevOps provides pre-defined CI/CD configuration which allows you to automatically detect, build, test,
|
||
deploy, and monitor your applications. Leveraging CI/CD best practices and tools, Auto DevOps aims
|
||
to simplify the setup and execution of a mature & modern software development lifecycle.
|
||
|
||
## Overview
|
||
|
||
With Auto DevOps, the software development process becomes easier to set up
|
||
as every project can have a complete workflow from verification to monitoring
|
||
with minimal configuration. Just push your code and GitLab takes
|
||
care of everything else. This makes it easier to start new projects and brings
|
||
consistency to how applications are set up throughout a company.
|
||
|
||
For an introduction to Auto DevOps, watch [AutoDevOps in GitLab 11.0](https://youtu.be/0Tc0YYBxqi4).
|
||
|
||
## Enabled by default
|
||
|
||
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/41729) in GitLab 11.3.
|
||
|
||
Auto DevOps is enabled by default for all projects and will attempt to run on all pipelines
|
||
in each project. This default can be enabled or disabled by an instance administrator in the
|
||
[Auto DevOps settings](../../user/admin_area/settings/continuous_integration.md#auto-devops-core-only).
|
||
It will be automatically disabled in individual projects on their first pipeline failure,
|
||
if it has not been explicitly enabled for the project.
|
||
|
||
Since [GitLab 12.7](https://gitlab.com/gitlab-org/gitlab/issues/26655), Auto DevOps
|
||
will run on pipelines automatically only if a [`Dockerfile` or matching buildpack](stages.md#auto-build)
|
||
exists.
|
||
|
||
If a [CI/CD configuration file](../../ci/yaml/README.md) is present in the project,
|
||
it will continue to be used, whether or not Auto DevOps is enabled.
|
||
|
||
## Quick start
|
||
|
||
If you are using GitLab.com, see the [quick start guide](quick_start_guide.md)
|
||
for how to use Auto DevOps with GitLab.com and a Kubernetes cluster on Google Kubernetes
|
||
Engine (GKE).
|
||
|
||
If you are using a self-managed instance of GitLab, you will need to configure the
|
||
[Google OAuth2 OmniAuth Provider](../../integration/google.md) before
|
||
you can configure a cluster on GKE. Once this is set up, you can follow the steps on the
|
||
[quick start guide](quick_start_guide.md) to get started.
|
||
|
||
## Comparison to application platforms and PaaS
|
||
|
||
Auto DevOps provides functionality that is often included in an application
|
||
platform or a Platform as a Service (PaaS). It takes inspiration from the
|
||
innovative work done by [Heroku](https://www.heroku.com/) and goes beyond it
|
||
in multiple ways:
|
||
|
||
- Auto DevOps works with any Kubernetes cluster; you're not limited to running
|
||
on GitLab's infrastructure. (Note that many features also work without Kubernetes).
|
||
- There is no additional cost (no markup on the infrastructure costs), and you
|
||
can use a Kubernetes cluster you host or Containers as a Service on any
|
||
public cloud (for example, [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/)).
|
||
- Auto DevOps has more features including security testing, performance testing,
|
||
and code quality testing.
|
||
- Auto DevOps offers an incremental graduation path. If you need advanced customizations,
|
||
you can start modifying the templates without having to start over on a
|
||
completely different platform. Review the [customizing](customize.md) documentation for more information.
|
||
|
||
## Features
|
||
|
||
Comprised of a set of stages, Auto DevOps brings these best practices to your
|
||
project in a simple and automatic way:
|
||
|
||
1. [Auto Build](stages.md#auto-build)
|
||
1. [Auto Test](stages.md#auto-test)
|
||
1. [Auto Code Quality](stages.md#auto-code-quality-starter) **(STARTER)**
|
||
1. [Auto SAST (Static Application Security Testing)](stages.md#auto-sast-ultimate) **(ULTIMATE)**
|
||
1. [Auto Dependency Scanning](stages.md#auto-dependency-scanning-ultimate) **(ULTIMATE)**
|
||
1. [Auto License Compliance](stages.md#auto-license-compliance-ultimate) **(ULTIMATE)**
|
||
1. [Auto Container Scanning](stages.md#auto-container-scanning-ultimate) **(ULTIMATE)**
|
||
1. [Auto Review Apps](stages.md#auto-review-apps)
|
||
1. [Auto DAST (Dynamic Application Security Testing)](stages.md#auto-dast-ultimate) **(ULTIMATE)**
|
||
1. [Auto Deploy](stages.md#auto-deploy)
|
||
1. [Auto Browser Performance Testing](stages.md#auto-browser-performance-testing-premium) **(PREMIUM)**
|
||
1. [Auto Monitoring](stages.md#auto-monitoring)
|
||
|
||
As Auto DevOps relies on many different components, it's good to have a basic
|
||
knowledge of the following:
|
||
|
||
- [Kubernetes](https://kubernetes.io/docs/home/)
|
||
- [Helm](https://helm.sh/docs/)
|
||
- [Docker](https://docs.docker.com)
|
||
- [GitLab Runner](https://docs.gitlab.com/runner/)
|
||
- [Prometheus](https://prometheus.io/docs/introduction/overview/)
|
||
|
||
Auto DevOps provides great defaults for all the stages; you can, however,
|
||
[customize](customize.md) almost everything to your needs.
|
||
|
||
For an overview on the creation of Auto DevOps, read more
|
||
[in this blog post](https://about.gitlab.com/blog/2017/06/29/whats-next-for-gitlab-ci/).
|
||
|
||
NOTE: **Note**
|
||
Kubernetes clusters can [be used without](../../user/project/clusters/index.md)
|
||
Auto DevOps.
|
||
|
||
## Requirements
|
||
|
||
To make full use of Auto DevOps, you will need:
|
||
|
||
- **Kubernetes** (for Auto Review Apps, Auto Deploy, and Auto Monitoring)
|
||
|
||
To enable deployments, you will need:
|
||
|
||
1. A [Kubernetes 1.12+ cluster](../../user/project/clusters/index.md) for the project. The easiest
|
||
way is to create a [new cluster using the GitLab UI](../../user/project/clusters/add_remove_clusters.md#create-new-cluster).
|
||
For Kubernetes 1.16+ clusters, there is some additional configuration for [Auto Deploy for Kubernetes 1.16+](stages.md#kubernetes-116).
|
||
1. NGINX Ingress. You can deploy it to your Kubernetes cluster by installing
|
||
the [GitLab-managed app for Ingress](../../user/clusters/applications.md#ingress),
|
||
once you have configured GitLab's Kubernetes integration in the previous step.
|
||
|
||
Alternatively, you can use the
|
||
[`nginx-ingress`](https://github.com/helm/charts/tree/master/stable/nginx-ingress)
|
||
Helm chart to install Ingress manually.
|
||
|
||
NOTE: **Note:**
|
||
If you are using your own Ingress instead of the one provided by GitLab's managed
|
||
apps, ensure you are running at least version 0.9.0 of NGINX Ingress and
|
||
[enable Prometheus metrics](https://github.com/helm/charts/tree/master/stable/nginx-ingress#prometheus-metrics)
|
||
in order for the response metrics to appear. You will also have to
|
||
[annotate](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/)
|
||
the NGINX Ingress deployment to be scraped by Prometheus using
|
||
`prometheus.io/scrape: "true"` and `prometheus.io/port: "10254"`.
|
||
|
||
- **Base domain** (for Auto Review Apps, Auto Deploy, and Auto Monitoring)
|
||
|
||
You will need a domain configured with wildcard DNS which is going to be used
|
||
by all of your Auto DevOps applications. If you're using the
|
||
[GitLab-managed app for Ingress](../../user/clusters/applications.md#ingress),
|
||
the URL endpoint will be automatically configured for you.
|
||
|
||
You will then need to [specify the Auto DevOps base domain](#auto-devops-base-domain).
|
||
|
||
- **GitLab Runner** (for all stages)
|
||
|
||
Your Runner needs to be configured to be able to run Docker. Generally this
|
||
means using either the [Docker](https://docs.gitlab.com/runner/executors/docker.html)
|
||
or [Kubernetes](https://docs.gitlab.com/runner/executors/kubernetes.html) executors, with
|
||
[privileged mode enabled](https://docs.gitlab.com/runner/executors/docker.html#use-docker-in-docker-with-privileged-mode).
|
||
The Runners do not need to be installed in the Kubernetes cluster, but the
|
||
Kubernetes executor is easy to use and is automatically autoscaling.
|
||
Docker-based Runners can be configured to autoscale as well, using [Docker
|
||
Machine](https://docs.gitlab.com/runner/install/autoscaling.html).
|
||
|
||
If you have configured GitLab's Kubernetes integration in the first step, you
|
||
can deploy it to your cluster by installing the
|
||
[GitLab-managed app for GitLab Runner](../../user/clusters/applications.md#gitlab-runner).
|
||
|
||
Runners should be registered as [shared Runners](../../ci/runners/README.md#registering-a-shared-runner)
|
||
for the entire GitLab instance, or [specific Runners](../../ci/runners/README.md#registering-a-specific-runner)
|
||
that are assigned to specific projects (the default if you have installed the
|
||
GitLab Runner managed application).
|
||
|
||
- **Prometheus** (for Auto Monitoring)
|
||
|
||
To enable Auto Monitoring, you will need Prometheus installed somewhere
|
||
(inside or outside your cluster) and configured to scrape your Kubernetes cluster.
|
||
If you have configured GitLab's Kubernetes integration, you can deploy it to
|
||
your cluster by installing the
|
||
[GitLab-managed app for Prometheus](../../user/clusters/applications.md#prometheus).
|
||
|
||
The [Prometheus service](../../user/project/integrations/prometheus.md)
|
||
integration needs to be enabled for the project (or enabled as a
|
||
[default service template](../../user/project/integrations/services_templates.md)
|
||
for the entire GitLab installation).
|
||
|
||
To get response metrics (in addition to system metrics), you need to
|
||
[configure Prometheus to monitor NGINX](../../user/project/integrations/prometheus_library/nginx_ingress.md#configuring-nginx-ingress-monitoring).
|
||
|
||
- **cert-manager** (optional, for TLS/HTTPS)
|
||
|
||
To enable HTTPS endpoints for your application, you need to install cert-manager,
|
||
a native Kubernetes certificate management controller that helps with issuing certificates.
|
||
Installing cert-manager on your cluster will issue a certificate by
|
||
[Let’s Encrypt](https://letsencrypt.org/) and ensure that certificates are valid and up-to-date.
|
||
If you have configured GitLab's Kubernetes integration, you can deploy it to
|
||
your cluster by installing the
|
||
[GitLab-managed app for cert-manager](../../user/clusters/applications.md#cert-manager).
|
||
|
||
If you do not have Kubernetes or Prometheus installed, then Auto Review Apps,
|
||
Auto Deploy, and Auto Monitoring will be silently skipped.
|
||
|
||
One all requirements are met, you can go ahead and [enable Auto DevOps](#enablingdisabling-auto-devops).
|
||
|
||
## Auto DevOps base domain
|
||
|
||
The Auto DevOps base domain is required if you want to make use of
|
||
[Auto Review Apps](stages.md#auto-review-apps), [Auto Deploy](stages.md#auto-deploy), and
|
||
[Auto Monitoring](stages.md#auto-monitoring). It can be defined in any of the following
|
||
places:
|
||
|
||
- either under the cluster's settings, whether for [projects](../../user/project/clusters/index.md#base-domain) or [groups](../../user/group/clusters/index.md#base-domain)
|
||
- or in instance-wide settings in the **Admin Area > Settings** under the "Continuous Integration and Delivery" section
|
||
- or at the project level as a variable: `KUBE_INGRESS_BASE_DOMAIN`
|
||
- or at the group level as a variable: `KUBE_INGRESS_BASE_DOMAIN`.
|
||
|
||
The base domain variable `KUBE_INGRESS_BASE_DOMAIN` follows the same order of precedence
|
||
as other environment [variables](../../ci/variables/README.md#priority-of-environment-variables).
|
||
|
||
TIP: **Tip:**
|
||
If you're using the [GitLab managed app for Ingress](../../user/clusters/applications.md#ingress),
|
||
the URL endpoint should be automatically configured for you. All you have to do
|
||
is use its value for the `KUBE_INGRESS_BASE_DOMAIN` variable.
|
||
|
||
NOTE: **Note:**
|
||
`AUTO_DEVOPS_DOMAIN` was [deprecated in GitLab 11.8](https://gitlab.com/gitlab-org/gitlab-foss/issues/52363)
|
||
and replaced with `KUBE_INGRESS_BASE_DOMAIN`. It was removed in
|
||
[GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-foss/issues/56959).
|
||
|
||
A wildcard DNS A record matching the base domain(s) is required, for example,
|
||
given a base domain of `example.com`, you'd need a DNS entry like:
|
||
|
||
```text
|
||
*.example.com 3600 A 1.2.3.4
|
||
```
|
||
|
||
In this case, `example.com` is the domain name under which the deployed apps will be served,
|
||
and `1.2.3.4` is the IP address of your load balancer; generally NGINX
|
||
([see requirements](#requirements)). How to set up the DNS record is beyond
|
||
the scope of this document; you should check with your DNS provider.
|
||
|
||
Alternatively you can use free public services like [nip.io](https://nip.io)
|
||
which provide automatic wildcard DNS without any configuration. Just set the
|
||
Auto DevOps base domain to `1.2.3.4.nip.io`.
|
||
|
||
Once set up, all requests will hit the load balancer, which in turn will route
|
||
them to the Kubernetes pods that run your application(s).
|
||
|
||
## Enabling/Disabling Auto DevOps
|
||
|
||
When first using Auto DevOps, review the [requirements](#requirements) to ensure all necessary components to make
|
||
full use of Auto DevOps are available. If this is your fist time, we recommend you follow the
|
||
[quick start guide](quick_start_guide.md).
|
||
|
||
GitLab.com users can enable/disable Auto DevOps at the project-level only. Self-managed users
|
||
can enable/disable Auto DevOps at the project-level, group-level or instance-level.
|
||
|
||
### At the project level
|
||
|
||
If enabling, check that your project doesn't have a `.gitlab-ci.yml`, or if one exists, remove it.
|
||
|
||
1. Go to your project's **Settings > CI/CD > Auto DevOps**.
|
||
1. Toggle the **Default to Auto DevOps pipeline** checkbox (checked to enable, unchecked to disable)
|
||
1. When enabling, it's optional but recommended to add in the [base domain](#auto-devops-base-domain)
|
||
that will be used by Auto DevOps to [deploy your application](stages.md#auto-deploy)
|
||
and choose the [deployment strategy](#deployment-strategy).
|
||
1. Click **Save changes** for the changes to take effect.
|
||
|
||
When the feature has been enabled, an Auto DevOps pipeline is triggered on the default branch.
|
||
|
||
### At the group level
|
||
|
||
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/52447) in GitLab 11.10.
|
||
|
||
Only administrators and group owners can enable or disable Auto DevOps at the group level.
|
||
|
||
To enable or disable Auto DevOps at the group-level:
|
||
|
||
1. Go to group's **Settings > CI/CD > Auto DevOps** page.
|
||
1. Toggle the **Default to Auto DevOps pipeline** checkbox (checked to enable, unchecked to disable).
|
||
1. Click **Save changes** button for the changes to take effect.
|
||
|
||
When enabling or disabling Auto DevOps at group-level, group configuration will be implicitly used for
|
||
the subgroups and projects inside that group, unless Auto DevOps is specifically enabled or disabled on
|
||
the subgroup or project.
|
||
|
||
### At the instance level (Administrators only)
|
||
|
||
Even when disabled at the instance level, group owners and project maintainers can still enable
|
||
Auto DevOps at the group and project level, respectively.
|
||
|
||
1. Go to **Admin Area > Settings > Continuous Integration and Deployment**.
|
||
1. Toggle the checkbox labeled **Default to Auto DevOps pipeline for all projects**.
|
||
1. If enabling, optionally set up the Auto DevOps [base domain](#auto-devops-base-domain) which will be used for Auto Deploy and Auto Review Apps.
|
||
1. Click **Save changes** for the changes to take effect.
|
||
|
||
### Enable for a percentage of projects
|
||
|
||
There is also a feature flag to enable Auto DevOps by default to your chosen percentage of projects.
|
||
|
||
This can be enabled from the console with the following, which uses the example of 10%:
|
||
|
||
`Feature.get(:force_autodevops_on_by_default).enable_percentage_of_actors(10)`
|
||
|
||
### Deployment strategy
|
||
|
||
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/38542) in GitLab 11.0.
|
||
|
||
You can change the deployment strategy used by Auto DevOps by going to your
|
||
project's **Settings > CI/CD > Auto DevOps**.
|
||
|
||
The available options are:
|
||
|
||
- **Continuous deployment to production**: Enables [Auto Deploy](stages.md#auto-deploy)
|
||
with `master` branch directly deployed to production.
|
||
- **Continuous deployment to production using timed incremental rollout**: Sets the
|
||
[`INCREMENTAL_ROLLOUT_MODE`](customize.md#timed-incremental-rollout-to-production-premium) variable
|
||
to `timed`, and production deployment will be executed with a 5 minute delay between
|
||
each increment in rollout.
|
||
- **Automatic deployment to staging, manual deployment to production**: Sets the
|
||
[`STAGING_ENABLED`](customize.md#deploy-policy-for-staging-and-production-environments) and
|
||
[`INCREMENTAL_ROLLOUT_MODE`](customize.md#incremental-rollout-to-production-premium) variables
|
||
to `1` and `manual`. This means:
|
||
|
||
- `master` branch is directly deployed to staging.
|
||
- Manual actions are provided for incremental rollout to production.
|
||
|
||
## Using multiple Kubernetes clusters **(PREMIUM)**
|
||
|
||
When using Auto DevOps, you may want to deploy different environments to
|
||
different Kubernetes clusters. This is possible due to the 1:1 connection that
|
||
[exists between them](../../user/project/clusters/index.md#multiple-kubernetes-clusters-premium).
|
||
|
||
In the [Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) (used behind the scenes by Auto DevOps), there
|
||
are currently 3 defined environment names that you need to know:
|
||
|
||
- `review/` (every environment starting with `review/`)
|
||
- `staging`
|
||
- `production`
|
||
|
||
Those environments are tied to jobs that use [Auto Deploy](stages.md#auto-deploy), so
|
||
except for the environment scope, they would also need to have a different
|
||
domain they would be deployed to. This is why you need to define a separate
|
||
`KUBE_INGRESS_BASE_DOMAIN` variable for all the above
|
||
[based on the environment](../../ci/variables/README.md#limiting-environment-scopes-of-environment-variables).
|
||
|
||
The following table is an example of how the three different clusters would
|
||
be configured.
|
||
|
||
| Cluster name | Cluster environment scope | `KUBE_INGRESS_BASE_DOMAIN` variable value | Variable environment scope | Notes |
|
||
|--------------|---------------------------|-------------------------------------------|----------------------------|---|
|
||
| review | `review/*` | `review.example.com` | `review/*` | The review cluster which will run all [Review Apps](../../ci/review_apps/index.md). `*` is a wildcard, which means it will be used by every environment name starting with `review/`. |
|
||
| staging | `staging` | `staging.example.com` | `staging` | (Optional) The staging cluster which will run the deployments of the staging environments. You need to [enable it first](customize.md#deploy-policy-for-staging-and-production-environments). |
|
||
| production | `production` | `example.com` | `production` | The production cluster which will run the deployments of the production environment. You can use [incremental rollouts](customize.md#incremental-rollout-to-production-premium). |
|
||
|
||
To add a different cluster for each environment:
|
||
|
||
1. Navigate to your project's **Operations > Kubernetes** and create the Kubernetes clusters
|
||
with their respective environment scope as described from the table above.
|
||
|
||
|
||
|
||
1. After the clusters are created, navigate to each one and install Helm Tiller
|
||
and Ingress. Wait for the Ingress IP address to be assigned.
|
||
1. Make sure you have [configured your DNS](#auto-devops-base-domain) with the
|
||
specified Auto DevOps domains.
|
||
1. Navigate to each cluster's page, through **Operations > Kubernetes**,
|
||
and add the domain based on its Ingress IP address.
|
||
|
||
Now that all is configured, you can test your setup by creating a merge request
|
||
and verifying that your app is deployed as a review app in the Kubernetes
|
||
cluster with the `review/*` environment scope. Similarly, you can check the
|
||
other environments.
|
||
|
||
## Currently supported languages
|
||
|
||
Note that not all buildpacks support Auto Test yet, as it's a relatively new
|
||
enhancement. All of Heroku's [officially supported
|
||
languages](https://devcenter.heroku.com/articles/heroku-ci#currently-supported-languages)
|
||
support it, and some third-party buildpacks as well e.g., Go, Node, Java, PHP,
|
||
Python, Ruby, Gradle, Scala, and Elixir all support Auto Test, but notably the
|
||
multi-buildpack does not.
|
||
|
||
As of GitLab 10.0, the supported buildpacks are:
|
||
|
||
```text
|
||
- heroku-buildpack-multi v1.0.0
|
||
- heroku-buildpack-ruby v168
|
||
- heroku-buildpack-nodejs v99
|
||
- heroku-buildpack-clojure v77
|
||
- heroku-buildpack-python v99
|
||
- heroku-buildpack-java v53
|
||
- heroku-buildpack-gradle v23
|
||
- heroku-buildpack-scala v78
|
||
- heroku-buildpack-play v26
|
||
- heroku-buildpack-php v122
|
||
- heroku-buildpack-go v72
|
||
- heroku-buildpack-erlang fa17af9
|
||
- buildpack-nginx v8
|
||
```
|
||
|
||
## Limitations
|
||
|
||
The following restrictions apply.
|
||
|
||
### Private registry support
|
||
|
||
There is no documented way of using private container registry with Auto DevOps.
|
||
We strongly advise using GitLab Container Registry with Auto DevOps in order to
|
||
simplify configuration and prevent any unforeseen issues.
|
||
|
||
### Installing Helm behind a proxy
|
||
|
||
GitLab does not yet support installing [Helm as a GitLab-managed App](../../user/clusters/applications.md#helm) when
|
||
behind a proxy. Users who wish to do so must inject their proxy settings
|
||
into the installation pods at runtime, for example by using a
|
||
[`PodPreset`](https://kubernetes.io/docs/concepts/workloads/pods/podpreset/):
|
||
|
||
```yml
|
||
apiVersion: settings.k8s.io/v1alpha1
|
||
kind: PodPreset
|
||
metadata:
|
||
name: gitlab-managed-apps-default-proxy
|
||
namespace: gitlab-managed-apps
|
||
spec:
|
||
env:
|
||
- name: http_proxy
|
||
value: "PUT_YOUR_HTTP_PROXY_HERE"
|
||
- name: https_proxy
|
||
value: "PUT_YOUR_HTTPS_PROXY_HERE"
|
||
```
|
||
|
||
## Troubleshooting
|
||
|
||
- Auto Build and Auto Test may fail to detect your language or framework with the
|
||
following error:
|
||
|
||
```plaintext
|
||
Step 5/11 : RUN /bin/herokuish buildpack build
|
||
---> Running in eb468cd46085
|
||
-----> Unable to select a buildpack
|
||
The command '/bin/sh -c /bin/herokuish buildpack build' returned a non-zero code: 1
|
||
```
|
||
|
||
The following are possible reasons:
|
||
|
||
- Your application may be missing the key files the buildpack is looking for. For
|
||
example, for Ruby applications you must have a `Gemfile` to be properly detected,
|
||
even though it is possible to write a Ruby app without a `Gemfile`.
|
||
- There may be no buildpack for your application. Try specifying a
|
||
[custom buildpack](customize.md#custom-buildpacks).
|
||
- Auto Test may fail because of a mismatch between testing frameworks. In this
|
||
case, you may need to customize your `.gitlab-ci.yml` with your test commands.
|
||
- Auto Deploy will fail if GitLab can not create a Kubernetes namespace and
|
||
service account for your project. For help debugging this issue, see
|
||
[Troubleshooting failed deployment jobs](../../user/project/clusters/index.md#troubleshooting).
|
||
|
||
## Development guides
|
||
|
||
[Development guide for Auto DevOps](../../development/auto_devops.md)
|