247 lines
10 KiB
Markdown
247 lines
10 KiB
Markdown
---
|
||
stage: Configure
|
||
group: Configure
|
||
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
|
||
---
|
||
|
||
# Troubleshooting Auto DevOps **(FREE)**
|
||
|
||
The information in this documentation page describes common errors when using
|
||
Auto DevOps, and any available workarounds.
|
||
|
||
## Unable to select a buildpack
|
||
|
||
Auto Build and Auto Test may fail to detect your language or framework with the
|
||
following error:
|
||
|
||
```plaintext
|
||
Step 5/11 : RUN /bin/herokuish buildpack build
|
||
---> Running in eb468cd46085
|
||
-----> Unable to select a buildpack
|
||
The command '/bin/sh -c /bin/herokuish buildpack build' returned a non-zero code: 1
|
||
```
|
||
|
||
The following are possible reasons:
|
||
|
||
- Your application may be missing the key files the buildpack is looking for.
|
||
Ruby applications require a `Gemfile` to be properly detected,
|
||
even though it's possible to write a Ruby app without a `Gemfile`.
|
||
- No buildpack may exist for your application. Try specifying a
|
||
[custom buildpack](customize.md#custom-buildpacks).
|
||
|
||
## Pipeline that extends Auto DevOps with only / except fails
|
||
|
||
If your pipeline fails with the following message:
|
||
|
||
```plaintext
|
||
Found errors in your .gitlab-ci.yml:
|
||
|
||
jobs:test config key may not be used with `rules`: only
|
||
```
|
||
|
||
This error appears when the included job’s rules configuration has been overridden with the `only` or `except` syntax.
|
||
To fix this issue, you must either:
|
||
|
||
- Transition your `only/except` syntax to rules.
|
||
- (Temporarily) Pin your templates to the [GitLab 12.10 based templates](https://gitlab.com/gitlab-org/auto-devops-v12-10).
|
||
|
||
## Failure to create a Kubernetes namespace
|
||
|
||
Auto Deploy fails if GitLab can't create a Kubernetes namespace and
|
||
service account for your project. For help debugging this issue, see
|
||
[Troubleshooting failed deployment jobs](../../user/project/clusters/index.md#troubleshooting).
|
||
|
||
## Detected an existing PostgreSQL database
|
||
|
||
After upgrading to GitLab 13.0, you may encounter this message when deploying
|
||
with Auto DevOps:
|
||
|
||
```plaintext
|
||
Detected an existing PostgreSQL database installed on the
|
||
deprecated channel 1, but the current channel is set to 2. The default
|
||
channel changed to 2 in of GitLab 13.0.
|
||
[...]
|
||
```
|
||
|
||
Auto DevOps, by default, installs an in-cluster PostgreSQL database alongside
|
||
your application. The default installation method changed in GitLab 13.0, and
|
||
upgrading existing databases requires user involvement. The two installation
|
||
methods are:
|
||
|
||
- **channel 1 (deprecated):** Pulls in the database as a dependency of the associated
|
||
Helm chart. Only supports Kubernetes versions up to version 1.15.
|
||
- **channel 2 (current):** Installs the database as an independent Helm chart. Required
|
||
for using the in-cluster database feature with Kubernetes versions 1.16 and greater.
|
||
|
||
If you receive this error, you can do one of the following actions:
|
||
|
||
- You can *safely* ignore the warning and continue using the channel 1 PostgreSQL
|
||
database by setting `AUTO_DEVOPS_POSTGRES_CHANNEL` to `1` and redeploying.
|
||
|
||
- You can delete the channel 1 PostgreSQL database and install a fresh channel 2
|
||
database by setting `AUTO_DEVOPS_POSTGRES_DELETE_V1` to a non-empty value and
|
||
redeploying.
|
||
|
||
WARNING:
|
||
Deleting the channel 1 PostgreSQL database permanently deletes the existing
|
||
channel 1 database and all its data. See
|
||
[Upgrading PostgreSQL](upgrading_postgresql.md)
|
||
for more information on backing up and upgrading your database.
|
||
|
||
- If you are not using the in-cluster database, you can set
|
||
`POSTGRES_ENABLED` to `false` and re-deploy. This option is especially relevant to
|
||
users of *custom charts without the in-chart PostgreSQL dependency*.
|
||
Database auto-detection is based on the `postgresql.enabled` Helm value for
|
||
your release. This value is set based on the `POSTGRES_ENABLED` CI/CD variable
|
||
and persisted by Helm, regardless of whether or not your chart uses the
|
||
variable.
|
||
|
||
WARNING:
|
||
Setting `POSTGRES_ENABLED` to `false` permanently deletes any existing
|
||
channel 1 database for your environment.
|
||
|
||
## Error: unable to recognize "": no matches for kind "Deployment" in version "extensions/v1beta1"
|
||
|
||
After upgrading your Kubernetes cluster to [v1.16+](stages.md#kubernetes-116),
|
||
you may encounter this message when deploying with Auto DevOps:
|
||
|
||
```plaintext
|
||
UPGRADE FAILED
|
||
Error: failed decoding reader into objects: unable to recognize "": no matches for kind "Deployment" in version "extensions/v1beta1"
|
||
```
|
||
|
||
This can occur if your current deployments on the environment namespace were deployed with a
|
||
deprecated/removed API that doesn't exist in Kubernetes v1.16+. For example,
|
||
if [your in-cluster PostgreSQL was installed in a legacy way](#detected-an-existing-postgresql-database),
|
||
the resource was created via the `extensions/v1beta1` API. However, the deployment resource
|
||
was moved to the `app/v1` API in v1.16.
|
||
|
||
To recover such outdated resources, you must convert the current deployments by mapping legacy APIs
|
||
to newer APIs. There is a helper tool called [`mapkubeapis`](https://github.com/hickeyma/helm-mapkubeapis)
|
||
that works for this problem. Follow these steps to use the tool in Auto DevOps:
|
||
|
||
1. Modify your `.gitlab-ci.yml` with:
|
||
|
||
```yaml
|
||
include:
|
||
- template: Auto-DevOps.gitlab-ci.yml
|
||
- remote: https://gitlab.com/shinya.maeda/ci-templates/-/raw/master/map-deprecated-api.gitlab-ci.yml
|
||
|
||
variables:
|
||
HELM_VERSION_FOR_MAPKUBEAPIS: "v2" # If you're using auto-depoy-image v2 or above, please specify "v3".
|
||
```
|
||
|
||
1. Run the job `<environment-name>:map-deprecated-api`. Ensure that this job succeeds before moving
|
||
to the next step. You should see something like the following output:
|
||
|
||
```shell
|
||
2020/10/06 07:20:49 Found deprecated or removed Kubernetes API:
|
||
"apiVersion: extensions/v1beta1
|
||
kind: Deployment"
|
||
Supported API equivalent:
|
||
"apiVersion: apps/v1
|
||
kind: Deployment"
|
||
```
|
||
|
||
1. Revert your `.gitlab-ci.yml` to the previous version. You no longer need to include the
|
||
supplemental template `map-deprecated-api`.
|
||
|
||
1. Continue the deployments as usual.
|
||
|
||
## Error: error initializing: Looks like "https://kubernetes-charts.storage.googleapis.com" is not a valid chart repository or cannot be reached
|
||
|
||
As [announced in the official CNCF blog post](https://www.cncf.io/blog/2020/10/07/important-reminder-for-all-helm-users-stable-incubator-repos-are-deprecated-and-all-images-are-changing-location/),
|
||
the stable Helm chart repository was deprecated and removed on November 13th, 2020.
|
||
You may encounter this error after that date.
|
||
|
||
Some GitLab features had dependencies on the stable chart. To mitigate the impact, we changed them
|
||
to use new official repositories or the [Helm Stable Archive repository maintained by GitLab](https://gitlab.com/gitlab-org/cluster-integration/helm-stable-archive).
|
||
Auto Deploy contains [an example fix](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/merge_requests/127).
|
||
|
||
In Auto Deploy, `v1.0.6+` of `auto-deploy-image` no longer adds the deprecated stable repository to
|
||
the `helm` command. If you use a custom chart and it relies on the deprecated stable repository,
|
||
specify an older `auto-deploy-image` like this example:
|
||
|
||
```yaml
|
||
include:
|
||
- template: Auto-DevOps.gitlab-ci.yml
|
||
|
||
.auto-deploy:
|
||
image: "registry.gitlab.com/gitlab-org/cluster-integration/auto-deploy-image:v1.0.5"
|
||
```
|
||
|
||
Keep in mind that this approach stops working when the stable repository is removed,
|
||
so you must eventually fix your custom chart.
|
||
|
||
To fix your custom chart:
|
||
|
||
1. In your chart directory, update the `repository` value in your `requirements.yaml` file from :
|
||
|
||
```yaml
|
||
repository: "https://kubernetes-charts.storage.googleapis.com/"
|
||
```
|
||
|
||
to:
|
||
|
||
```yaml
|
||
repository: "https://charts.helm.sh/stable"
|
||
```
|
||
|
||
1. In your chart directory, run `helm dep update .` using the same Helm major version as Auto DevOps.
|
||
1. Commit the changes for the `requirements.yaml` file.
|
||
1. If you previously had a `requirements.lock` file, commit the changes to the file.
|
||
If you did not previously have a `requirements.lock` file in your chart,
|
||
you do not need to commit the new one. This file is optional, but when present,
|
||
it's used to verify the integrity of the downloaded dependencies.
|
||
|
||
You can find more information in
|
||
[issue #263778, "Migrate PostgreSQL from stable Helm repository"](https://gitlab.com/gitlab-org/gitlab/-/issues/263778).
|
||
|
||
## Error: release .... failed: timed out waiting for the condition
|
||
|
||
When getting started with Auto DevOps, you may encounter this error when first
|
||
deploying your application:
|
||
|
||
```plaintext
|
||
INSTALL FAILED
|
||
PURGING CHART
|
||
Error: release staging failed: timed out waiting for the condition
|
||
```
|
||
|
||
This is most likely caused by a failed liveness (or readiness) probe attempted
|
||
during the deployment process. By default, these probes are run against the root
|
||
page of the deployed application on port 5000. If your application isn't configured
|
||
to serve anything at the root page, or is configured to run on a specific port
|
||
*other* than 5000, this check fails.
|
||
|
||
If it fails, you should see these failures in the events for the relevant
|
||
Kubernetes namespace. These events look like the following example:
|
||
|
||
```plaintext
|
||
LAST SEEN TYPE REASON OBJECT MESSAGE
|
||
3m20s Warning Unhealthy pod/staging-85db88dcb6-rxd6g Readiness probe failed: Get http://10.192.0.6:5000/: dial tcp 10.192.0.6:5000: connect: connection refused
|
||
3m32s Warning Unhealthy pod/staging-85db88dcb6-rxd6g Liveness probe failed: Get http://10.192.0.6:5000/: dial tcp 10.192.0.6:5000: connect: connection refused
|
||
```
|
||
|
||
To change the port used for the liveness checks, pass
|
||
[custom values to the Helm chart](customize.md#customize-values-for-helm-chart)
|
||
used by Auto DevOps:
|
||
|
||
1. Create a directory and file at the root of your repository named `.gitlab/auto-deploy-values.yaml`.
|
||
|
||
1. Populate the file with the following content, replacing the port values with
|
||
the actual port number your application is configured to use:
|
||
|
||
```yaml
|
||
service:
|
||
internalPort: <port_value>
|
||
externalPort: <port_value>
|
||
```
|
||
|
||
1. Commit your changes.
|
||
|
||
After committing your changes, subsequent probes should use the newly-defined ports.
|
||
The page that's probed can also be changed by overriding the `livenessProbe.path`
|
||
and `readinessProbe.path` values (shown in the
|
||
[default `values.yaml`](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/blob/master/assets/auto-deploy-app/values.yaml)
|
||
file) in the same fashion.
|