debian-mirror-gitlab/doc/development/testing_guide/review_apps.md

287 lines
16 KiB
Markdown
Raw Normal View History

2021-01-29 00:20:46 +05:30
---
stage: none
group: unassigned
2021-02-22 17:27:13 +05:30
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
2021-01-29 00:20:46 +05:30
---
2019-02-15 15:39:39 +05:30
# Review Apps
2018-12-05 23:21:45 +05:30
2022-07-23 23:45:48 +05:30
Review Apps are deployed using the `start-review-app-pipeline` job which triggers a child pipeline containing a series of jobs to perform the various tasks needed to deploy a Review App.
2021-11-18 22:05:49 +05:30
![start-review-app-pipeline job](img/review-app-parent-pipeline.png)
For any of the following scenarios, the `start-review-app-pipeline` job would be automatically started:
2022-07-23 23:45:48 +05:30
- for merge requests with CI configuration changes
2021-11-18 22:05:49 +05:30
- for merge requests with frontend changes
2022-03-02 08:16:31 +05:30
- for merge requests with changes to `{,ee/,jh/}{app/controllers}/**/*`
2022-04-04 11:22:00 +05:30
- for merge requests with changes to `{,ee/,jh/}{app/models}/**/*`
2022-07-16 23:28:13 +05:30
- for merge requests with changes to `{,ee/,jh/}lib/{,ee/,jh/}gitlab/**/*`
2021-11-18 22:05:49 +05:30
- for merge requests with QA changes
- for scheduled pipelines
2021-12-11 22:18:48 +05:30
- the MR has the `pipeline:run-review-app` label set
2021-11-18 22:05:49 +05:30
## QA runs on Review Apps
On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage (which comes after the
2022-03-02 08:16:31 +05:30
`review` stage), the `review-qa-smoke` and `review-qa-reliable` jobs are automatically started. The `review-qa-smoke` runs
the QA smoke suite and the `review-qa-reliable` executes E2E tests identified as [reliable](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/reliable-tests).
2021-11-18 22:05:49 +05:30
2022-07-23 23:45:48 +05:30
`review-qa-*` jobs ensure that end-to-end tests for the changes in the merge request pass in a live environment. This shifts the identification of e2e failures from an environment on the path to production to the merge request, to prevent breaking features on GitLab.com or costly GitLab.com deployment blockers. `review-qa-*` failures should be investigated with counterpart SET involvement if needed to help determine the root cause of the error.
2021-11-18 22:05:49 +05:30
You can also manually start the `review-qa-all`: it runs the full QA suite.
After the end-to-end test runs have finished, [Allure reports](https://github.com/allure-framework/allure2) are generated and published by
2022-03-02 08:16:31 +05:30
the `allure-report-qa-smoke`, `allure-report-qa-reliable`, and `allure-report-qa-all` jobs. A comment with links to the reports are added to the merge request.
2021-11-18 22:05:49 +05:30
2022-04-04 11:22:00 +05:30
Errors can be found in the `gitlab-review-apps` Sentry project and [filterable by Review App URL](https://sentry.gitlab.net/gitlab/gitlab-review-apps/?query=url%3A%22https%3A%2F%2Fgitlab-review-require-ve-u92nn2.gitlab-review.app%2F%22) or [commit SHA](https://sentry.gitlab.net/gitlab/gitlab-review-apps/releases/6095b501da7/all-events/).
2022-07-23 23:45:48 +05:30
### Bypass failed review app deployment to merge a broken `master` fix
Maintainers can elect to use the [process for merging during broken `master`](https://about.gitlab.com/handbook/engineering/workflow/#instructions-for-the-maintainer) if a customer-critical merge request is blocked by pipelines failing due to review app deployment failures.
2021-11-18 22:05:49 +05:30
## Performance Metrics
On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage, the
`review-performance` job is automatically started: this job does basic
browser performance testing using a
2022-08-13 15:12:31 +05:30
[Sitespeed.io Container](../../ci/testing/browser_performance_testing.md).
2021-11-18 22:05:49 +05:30
2022-01-26 12:08:38 +05:30
## Sample Data for Review Apps
Upon deployment of a review app, project data is created from the [`sample-gitlab-project`](https://gitlab.com/gitlab-org/sample-data-templates/sample-gitlab-project) template project. This aims to provide projects with prepopulated resources to facilitate manual and exploratory testing.
The sample projects will be created in the `root` user namespace and can be accessed from the personal projects list for that user.
2021-11-18 22:05:49 +05:30
## How to
2021-12-11 22:18:48 +05:30
### Redeploy Review App from a clean slate
To reset Review App and redeploy from a clean slate, do the following:
1. Run `review-stop` job.
1. Re-deploy by running or retrying `review-deploy` job.
Doing this will remove all existing data from a previously deployed Review App.
2021-11-18 22:05:49 +05:30
### Get access to the GCP Review Apps cluster
You need to [open an access request (internal link)](https://gitlab.com/gitlab-com/access-requests/-/issues/new)
for the `gcp-review-apps-dev` GCP group and role.
This grants you the following permissions for:
- [Retrieving pod logs](#dig-into-a-pods-logs). Granted by [Viewer (`roles/viewer`)](https://cloud.google.com/iam/docs/understanding-roles#kubernetes-engine-roles).
- [Running a Rails console](#run-a-rails-console). Granted by [Kubernetes Engine Developer (`roles/container.pods.exec`)](https://cloud.google.com/iam/docs/understanding-roles#kubernetes-engine-roles).
### Log into my Review App
For GitLab Team Members only. If you want to sign in to the review app, review
the GitLab handbook information for the [shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams).
- The default username is `root`.
- The password can be found in the 1Password login item named `GitLab EE Review App`.
### Enable a feature flag for my Review App
1. Open your Review App and log in as documented above.
1. Create a personal access token.
1. Enable the feature flag using the [Feature flag API](../../api/features.md).
### Find my Review App slug
1. Open the `review-deploy` job.
1. Look for `** Deploying review-*`.
1. For instance for `** Deploying review-1234-abc-defg... **`,
your Review App slug would be `review-1234-abc-defg` in this case.
### Run a Rails console
1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.exec` permission first.
2022-01-26 12:08:38 +05:30
1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps). For example, `review-qa-raise-e-12chm0`.
1. Find and open the `toolbox` Deployment. For example, `review-qa-raise-e-12chm0-toolbox`.
2022-07-23 23:45:48 +05:30
1. Select the Pod in the "Managed pods" section. For example, `review-qa-raise-e-12chm0-toolbox-d5455cc8-2lsvz`.
1. Select the `KUBECTL` dropdown, then `Exec` -> `toolbox`.
2022-01-26 12:08:38 +05:30
1. Replace `-c toolbox -- ls` with `-it -- gitlab-rails console` from the
2021-11-18 22:05:49 +05:30
default command or
2022-01-26 12:08:38 +05:30
- Run `kubectl exec --namespace review-qa-raise-e-12chm0 review-qa-raise-e-12chm0-toolbox-d5455cc8-2lsvz -it -- gitlab-rails console` and
- Replace `review-qa-raise-e-12chm0-toolbox-d5455cc8-2lsvz`
2021-11-18 22:05:49 +05:30
with your Pod's name.
### Dig into a Pod's logs
1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.getLogs` permission first.
2022-01-26 12:08:38 +05:30
1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps). For example, `review-qa-raise-e-12chm0`.
1. Find and open the `migrations` Deployment. For example, `review-qa-raise-e-12chm0-migrations.1`.
2022-07-23 23:45:48 +05:30
1. Select the Pod in the "Managed pods" section. For example, `review-qa-raise-e-12chm0-migrations.1-nqwtx`.
1. Select `Container logs`.
2021-11-18 22:05:49 +05:30
Alternatively, you could use the [Logs Explorer](https://console.cloud.google.com/logs/query;query=?project=gitlab-review-apps) which provides more utility to search logs. An example query for a pod name is as follows:
```shell
resource.labels.pod_name:"review-qa-raise-e-12chm0-migrations"
```
2018-12-05 23:21:45 +05:30
## How does it work?
2019-07-31 22:56:46 +05:30
### CI/CD architecture diagram
2019-03-02 22:35:43 +05:30
2019-09-30 21:07:59 +05:30
```mermaid
2019-03-02 22:35:43 +05:30
graph TD
2020-07-28 23:09:34 +05:30
A["build-qa-image, compile-production-assets<br/>(canonical default refs only)"];
2020-04-08 14:13:33 +05:30
B[review-build-cng];
C[review-deploy];
D[CNG-mirror];
2022-03-02 08:16:31 +05:30
E[review-qa-smoke, review-qa-reliable];
2020-04-08 14:13:33 +05:30
A -->|once the `prepare` stage is done| B
B -.->|triggers a CNG-mirror pipeline and wait for it to be done| D
D -.->|polls until completed| B
B -->|once the `review-build-cng` job is done| C
C -->|once the `review-deploy` job is done| E
subgraph "1. gitlab `prepare` stage"
A
end
subgraph "2. gitlab `review-prepare` stage"
B
end
subgraph "3. gitlab `review` stage"
2020-07-28 23:09:34 +05:30
C["review-deploy<br><br>Helm deploys the Review App using the Cloud<br/>Native images built by the CNG-mirror pipeline.<br><br>Cloud Native images are deployed to the `review-apps`<br>Kubernetes (GKE) cluster, in the GCP `gitlab-review-apps` project."]
2020-04-08 14:13:33 +05:30
end
subgraph "4. gitlab `qa` stage"
2022-03-02 08:16:31 +05:30
E[review-qa-smoke, review-qa-reliable<br><br>gitlab-qa runs the smoke and reliable suites against the Review App.]
2020-04-08 14:13:33 +05:30
end
2019-07-31 22:56:46 +05:30
2019-09-30 21:07:59 +05:30
subgraph "CNG-mirror pipeline"
2020-04-08 14:13:33 +05:30
D>Cloud Native images are built];
end
2019-09-30 21:07:59 +05:30
```
2019-03-02 22:35:43 +05:30
### Detailed explanation
2020-07-28 23:09:34 +05:30
1. On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) during the `prepare` stage, the
[`compile-production-assets`](https://gitlab.com/gitlab-org/gitlab/-/jobs/641770154) job is automatically started.
- Once it's done, the [`review-build-cng`](https://gitlab.com/gitlab-org/gitlab/-/jobs/467724808)
job starts since the [`CNG-mirror`](https://gitlab.com/gitlab-org/build/CNG-mirror) pipeline triggered in the
2019-07-31 22:56:46 +05:30
following step depends on it.
2020-07-28 23:09:34 +05:30
1. Once `compile-production-assets` is done, the [`review-build-cng`](https://gitlab.com/gitlab-org/gitlab/-/jobs/467724808)
job [triggers a pipeline](https://gitlab.com/gitlab-org/build/CNG-mirror/pipelines/44364657)
2020-04-22 19:07:51 +05:30
in the [`CNG-mirror`](https://gitlab.com/gitlab-org/build/CNG-mirror) project.
2020-07-28 23:09:34 +05:30
- The `review-build-cng` job automatically starts only if your MR includes
[CI or frontend changes](../pipelines.md#changes-patterns). In other cases, the job is manual.
2020-04-22 19:07:51 +05:30
- The [`CNG-mirror`](https://gitlab.com/gitlab-org/build/CNG-mirror/pipelines/44364657) pipeline creates the Docker images of
2022-01-26 12:08:38 +05:30
each component (for example, `gitlab-rails-ee`, `gitlab-shell`, `gitaly` etc.)
2020-04-22 19:07:51 +05:30
based on the commit from the [GitLab pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) and stores
them in its [registry](https://gitlab.com/gitlab-org/build/CNG-mirror/container_registry).
2020-06-23 00:09:42 +05:30
- We use the [`CNG-mirror`](https://gitlab.com/gitlab-org/build/CNG-mirror) project so that the `CNG`, (Cloud
2020-07-28 23:09:34 +05:30
Native GitLab), project's registry is not overloaded with a lot of transient Docker images.
1. Once `review-build-cng` is done, the [`review-deploy`](https://gitlab.com/gitlab-org/gitlab/-/jobs/467724810) job
2020-04-22 19:07:51 +05:30
deploys the Review App using [the official GitLab Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/) to
2020-07-28 23:09:34 +05:30
the [`review-apps`](https://console.cloud.google.com/kubernetes/clusters/details/us-central1-b/review-apps?project=gitlab-review-apps)
2019-07-31 22:56:46 +05:30
Kubernetes cluster on GCP.
- The actual scripts used to deploy the Review App can be found at
2020-04-22 19:07:51 +05:30
[`scripts/review_apps/review-apps.sh`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/review_apps/review-apps.sh).
2019-07-31 22:56:46 +05:30
- These scripts are basically
2020-04-22 19:07:51 +05:30
[our official Auto DevOps scripts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) where the
2019-07-31 22:56:46 +05:30
default CNG images are overridden with the images built and stored in the
2020-04-22 19:07:51 +05:30
[`CNG-mirror` project's registry](https://gitlab.com/gitlab-org/build/CNG-mirror/container_registry).
- Since we're using [the official GitLab Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/), this means
2019-07-31 22:56:46 +05:30
you get a dedicated environment for your branch that's very close to what
it would look in production.
2021-09-04 01:27:46 +05:30
- Each review app is deployed to its own Kubernetes namespace. The namespace is based on the Review App slug that is
unique to each branch.
2020-04-22 19:07:51 +05:30
1. Once the [`review-deploy`](https://gitlab.com/gitlab-org/gitlab/-/jobs/467724810) job succeeds, you should be able to
2019-07-31 22:56:46 +05:30
use your Review App thanks to the direct link to it from the MR widget. To log
into the Review App, see "Log into my Review App?" below.
2018-12-05 23:21:45 +05:30
**Additional notes:**
2021-11-18 22:05:49 +05:30
- If the `review-deploy` job keeps failing (and a manual retry didn't help),
2021-12-11 22:18:48 +05:30
please post a message in the `#g_qe_engineering_productivity` channel and/or create a `~"Engineering Productivity"` `~"ep::review apps"` `~"type::bug"`
2019-09-04 21:01:54 +05:30
issue with a link to your merge request. Note that the deployment failure can
2022-01-26 12:08:38 +05:30
reveal an actual problem introduced in your merge request (that is, this isn't
2019-09-04 21:01:54 +05:30
necessarily a transient failure)!
2022-07-23 23:45:48 +05:30
- If the `review-qa-smoke` or `review-qa-reliable` job keeps failing (note that we already retry them once),
2019-09-04 21:01:54 +05:30
please check the job's logs: you could discover an actual problem introduced in
your merge request. You can also download the artifacts to see screenshots of
the page at the time the failures occurred. If you don't find the cause of the
failure or if it seems unrelated to your change, please post a message in the
2021-12-11 22:18:48 +05:30
`#quality` channel and/or create a ~Quality ~"type::bug" issue with a link to your
2019-09-04 21:01:54 +05:30
merge request.
2020-04-08 14:13:33 +05:30
- The manual `review-stop` can be used to
2019-07-31 22:56:46 +05:30
stop a Review App manually, and is also started by GitLab once a merge
request's branch is deleted after being merged.
2021-02-22 17:27:13 +05:30
- The Kubernetes cluster is connected to the `gitlab` projects using the
2021-11-18 22:05:49 +05:30
[GitLab Kubernetes integration](../../user/infrastructure/clusters/index.md). This basically
2020-07-28 23:09:34 +05:30
allows to have a link to the Review App directly from the merge request widget.
2019-02-15 15:39:39 +05:30
2020-04-08 14:13:33 +05:30
### Auto-stopping of Review Apps
Review Apps are automatically stopped 2 days after the last deployment thanks to
2021-04-17 20:07:23 +05:30
the [Environment auto-stop](../../ci/environments/index.md#stop-an-environment-after-a-certain-time-period) feature.
2020-04-08 14:13:33 +05:30
If you need your Review App to stay up for a longer time, you can
2021-04-17 20:07:23 +05:30
[pin its environment](../../ci/environments/index.md#override-a-deployments-scheduled-stop-time) or retry the
2020-04-08 14:13:33 +05:30
`review-deploy` job to update the "latest deployed at" time.
The `review-cleanup` job that automatically runs in scheduled
2022-06-21 17:19:12 +05:30
pipelines stops stale Review Apps after 5 days,
2020-04-08 14:13:33 +05:30
deletes their environment after 6 days, and cleans up any dangling Helm releases
and Kubernetes resources after 7 days.
2019-10-12 21:52:04 +05:30
## Cluster configuration
2021-09-04 01:27:46 +05:30
The cluster is configured via Terraform in the [`engineering-productivity-infrastructure`](https://gitlab.com/gitlab-org/quality/engineering-productivity-infrastructure) project.
2019-10-12 21:52:04 +05:30
2020-10-24 23:57:45 +05:30
Node pool image type must be `Container-Optimized OS (cos)`, not `Container-Optimized OS with Containerd (cos_containerd)`,
2022-01-26 12:08:38 +05:30
due to this [known issue on the Kubernetes executor for GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4755)
2020-10-24 23:57:45 +05:30
2020-04-22 19:07:51 +05:30
### Helm
2019-10-12 21:52:04 +05:30
2020-04-22 19:07:51 +05:30
The Helm version used is defined in the
2022-07-23 23:45:48 +05:30
[`registry.gitlab.com/gitlab-org/gitlab-build-images:gitlab-helm3.5-kubectl1.17` image](https://gitlab.com/gitlab-org/gitlab-build-images/-/blob/master/Dockerfile.gitlab-helm3.5-kubectl1.17#L6)
2020-01-01 13:55:28 +05:30
used by the `review-deploy` and `review-stop` jobs.
## Diagnosing unhealthy Review App releases
If [Review App Stability](https://app.periscopedata.com/app/gitlab/496118/Engineering-Productivity-Sandbox?widget=6690556&udv=785399)
2021-09-04 01:27:46 +05:30
dips this may be a signal that the `review-apps` cluster is unhealthy.
2020-04-22 19:07:51 +05:30
Leading indicators may be health check failures leading to restarts or majority failure for Review App deployments.
2020-01-01 13:55:28 +05:30
2020-06-23 00:09:42 +05:30
The [Review Apps Overview dashboard](https://console.cloud.google.com/monitoring/classic/dashboards/6798952013815386466?project=gitlab-review-apps&timeDomain=1d)
2020-01-01 13:55:28 +05:30
aids in identifying load spikes on the cluster, and if nodes are problematic or the entire cluster is trending towards unhealthy.
2022-06-21 17:19:12 +05:30
See the [review apps page of the Engineering Productivity Runbook](https://gitlab.com/gitlab-org/quality/engineering-productivity/team/-/blob/main/runbook/review-apps.md) for troubleshooting review app releases.
2019-09-04 21:01:54 +05:30
2018-12-05 23:21:45 +05:30
## Frequently Asked Questions
2019-02-15 15:39:39 +05:30
**Isn't it too much to trigger CNG image builds on every test run? This creates
thousands of unused Docker images.**
2018-12-05 23:21:45 +05:30
2019-02-15 15:39:39 +05:30
> We have to start somewhere and improve later. Also, we're using the
2019-12-04 20:38:33 +05:30
> CNG-mirror project to store these Docker images so that we can just wipe out
> the registry at some point, and use a new fresh, empty one.
2018-12-05 23:21:45 +05:30
2019-02-15 15:39:39 +05:30
**How do we secure this from abuse? Apps are open to the world so we need to
find a way to limit it to only us.**
2018-12-05 23:21:45 +05:30
2019-02-15 15:39:39 +05:30
> This isn't enabled for forks.
2018-12-05 23:21:45 +05:30
2019-07-07 11:18:12 +05:30
## Other resources
2019-09-30 21:07:59 +05:30
- [Review Apps integration for CE/EE (presentation)](https://docs.google.com/presentation/d/1QPLr6FO4LduROU8pQIPkX1yfGvD13GEJIBOenqoKxR8/edit?usp=sharing)
2020-06-23 00:09:42 +05:30
- [Stability issues](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/212)
2019-12-04 20:38:33 +05:30
### Helpful command line tools
2020-04-22 19:07:51 +05:30
- [K9s](https://github.com/derailed/k9s) - enables CLI dashboard across pods and enabling filtering by labels
2019-12-04 20:38:33 +05:30
- [Stern](https://github.com/wercker/stern) - enables cross pod log tailing based on label/field selectors
2019-07-07 11:18:12 +05:30
2018-12-05 23:21:45 +05:30
---
[Return to Testing documentation](index.md)