debian-mirror-gitlab/doc/administration/maintenance_mode/index.md

216 lines
10 KiB
Markdown
Raw Normal View History

2021-03-11 19:13:27 +05:30
---
2022-07-23 23:45:48 +05:30
stage: Systems
2021-03-11 19:13:27 +05:30
group: Geo
2022-11-25 23:54:43 +05:30
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
2021-03-11 19:13:27 +05:30
---
# GitLab Maintenance Mode **(PREMIUM SELF)**
2021-11-11 11:23:49 +05:30
> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2149) in GitLab 13.9.
2021-03-11 19:13:27 +05:30
2023-01-13 00:05:48 +05:30
Maintenance Mode allows administrators to reduce write operations to a minimum while maintenance tasks are performed. The main goal is to block all external actions that change the internal state, including the PostgreSQL database, but especially files, Git repositories, and Container repositories.
2021-03-11 19:13:27 +05:30
2023-01-13 00:05:48 +05:30
When Maintenance Mode is enabled, in-progress actions finish relatively quickly since no new actions are coming in, and internal state changes are minimal.
2021-03-11 19:13:27 +05:30
In that state, various maintenance tasks are easier, and services can be stopped completely or be
2021-06-08 01:23:25 +05:30
further degraded for a much shorter period of time than might otherwise be needed. For example, stopping cron jobs and draining queues should be fairly quick.
2021-03-11 19:13:27 +05:30
2022-07-23 23:45:48 +05:30
Maintenance Mode allows most external actions that do not change internal state. On a high-level, HTTP `POST`, `PUT`, `PATCH`, and `DELETE` requests are blocked and a detailed overview of [how special cases are handled](#rest-api) is available.
2021-03-11 19:13:27 +05:30
## Enable Maintenance Mode
There are three ways to enable Maintenance Mode as an administrator:
- **Web UI**:
2022-10-11 01:57:18 +05:30
1. On the top bar, select **Main menu > Admin**.
2021-09-04 01:27:46 +05:30
1. On the left sidebar, select **Settings > General**.
1. Expand **Maintenance Mode**, and toggle **Enable Maintenance Mode**.
2021-03-11 19:13:27 +05:30
You can optionally add a message for the banner as well.
2021-09-04 01:27:46 +05:30
1. Select **Save changes**.
2021-03-11 19:13:27 +05:30
- **API**:
```shell
curl --request PUT --header "PRIVATE-TOKEN:$ADMIN_TOKEN" "<gitlab-url>/api/v4/application/settings?maintenance_mode=true"
```
- [**Rails console**](../operations/rails_console.md#starting-a-rails-console-session):
```ruby
2021-09-04 01:27:46 +05:30
::Gitlab::CurrentSettings.update!(maintenance_mode: true)
::Gitlab::CurrentSettings.update!(maintenance_mode_message: "New message")
2021-03-11 19:13:27 +05:30
```
## Disable Maintenance Mode
There are three ways to disable Maintenance Mode:
- **Web UI**:
2022-10-11 01:57:18 +05:30
1. On the top bar, select **Main menu > Admin**.
2021-09-04 01:27:46 +05:30
1. On the left sidebar, select **Settings > General**.
1. Expand **Maintenance Mode**, and toggle **Enable Maintenance Mode**.
You can optionally add a message for the banner as well.
1. Select **Save changes**.
2021-03-11 19:13:27 +05:30
- **API**:
```shell
curl --request PUT --header "PRIVATE-TOKEN:$ADMIN_TOKEN" "<gitlab-url>/api/v4/application/settings?maintenance_mode=false"
```
- [**Rails console**](../operations/rails_console.md#starting-a-rails-console-session):
```ruby
2021-10-27 15:23:28 +05:30
::Gitlab::CurrentSettings.update!(maintenance_mode: false)
2021-03-11 19:13:27 +05:30
```
## Behavior of GitLab features in Maintenance Mode
When Maintenance Mode is enabled, a banner is displayed at the top of the page.
The banner can be customized with a specific message.
An error is displayed when a user tries to perform a write operation that isn't allowed.
2021-04-29 21:17:54 +05:30
![Maintenance Mode banner and error message](img/maintenance_mode_error_message.png)
2021-03-11 19:13:27 +05:30
NOTE:
In some cases, the visual feedback from an action could be misleading, for example when starring a project, the **Star** button changes to show the **Unstar** action, however, this is only the frontend update, and it doesn't take into account the failed status of the POST request. These visual bugs are to be fixed [in follow-up iterations](https://gitlab.com/gitlab-org/gitlab/-/issues/295197).
2021-11-18 22:05:49 +05:30
### Administrator functions
2021-03-11 19:13:27 +05:30
2021-06-08 01:23:25 +05:30
Systems administrators can edit the application settings. This allows
2021-03-11 19:13:27 +05:30
them to disable Maintenance Mode after it's been enabled.
### Authentication
2022-11-25 23:54:43 +05:30
All users can sign in and out of the GitLab instance but no new users can be created.
2021-03-11 19:13:27 +05:30
2021-06-08 01:23:25 +05:30
If there are [LDAP syncs](../auth/ldap/index.md) scheduled for that time, they fail since user creation is disabled. Similarly, [user creations based on SAML](../../integration/saml.md#general-setup) fail.
2021-03-11 19:13:27 +05:30
### Git actions
2021-06-08 01:23:25 +05:30
All read-only Git operations continue to work, for example
`git clone` and `git pull`. All write operations fail, both through the CLI and Web IDE with the error message: `Git push is not allowed because this GitLab instance is currently in (read-only) maintenance mode.`
2021-03-11 19:13:27 +05:30
2021-06-08 01:23:25 +05:30
If Geo is enabled, Git pushes to both primary and secondaries fail.
2021-03-11 19:13:27 +05:30
### Merge requests, issues, epics
2021-06-08 01:23:25 +05:30
All write actions except those mentioned above fail. For example, a user cannot update merge requests or issues.
2021-03-11 19:13:27 +05:30
### Incoming email
2021-06-08 01:23:25 +05:30
Creating new issue replies, issues (including new Service Desk issues), merge requests [by email](../incoming_email.md) fail.
2021-03-11 19:13:27 +05:30
### Outgoing email
2021-06-08 01:23:25 +05:30
Notification emails continue to arrive, but emails that require database writes, like resetting the password, do not arrive.
2021-03-11 19:13:27 +05:30
### REST API
2022-07-23 23:45:48 +05:30
For most JSON requests, `POST`, `PUT`, `PATCH`, and `DELETE` are blocked, and the API returns a 403 response with the error message: `You cannot perform write operations on a read-only instance`. Only the following requests are allowed:
2021-03-11 19:13:27 +05:30
|HTTP request | Allowed routes | Notes |
|:----:|:--------------------------------------:|:----:|
2022-07-23 23:45:48 +05:30
| `POST` | `/admin/application_settings/general` | To allow updating application settings in the administrator UI |
| `PUT` | `/api/v4/application/settings` | To allow updating application settings with the API |
2022-11-25 23:54:43 +05:30
| `POST` | `/users/sign_in` | To allow users to sign in. |
| `POST` | `/users/sign_out`| To allow users to sign out. |
| `POST` | `/oauth/token` | To allow users to sign in to a Geo secondary for the first time. |
2022-07-23 23:45:48 +05:30
| `POST` | `/admin/session`, `/admin/session/destroy` | To allow [Admin Mode for GitLab administrators](https://gitlab.com/groups/gitlab-org/-/epics/2158) |
| `POST` | Paths ending with `/compare`| Git revision routes. |
| `POST` | `.git/git-upload-pack` | To allow Git pull/clone. |
| `POST` | `/api/v4/internal` | [internal API routes](../../development/internal_api/index.md) |
| `POST` | `/admin/sidekiq` | To allow management of background jobs in the Admin Area |
| `POST` | `/admin/geo` | To allow updating Geo Nodes in the administrator UI |
| `POST` | `/api/v4/geo_replication`| To allow certain Geo-specific administrator UI actions on secondary sites |
2021-03-11 19:13:27 +05:30
### GraphQL API
`POST /api/graphql` requests are allowed but mutations are blocked with the error message `You cannot perform write operations on a read-only instance`.
### Continuous Integration
- No new jobs or pipelines start, scheduled or otherwise.
- Jobs that were already running continue to have a `running` status in the GitLab UI,
even if they finish running on the GitLab Runner.
- Jobs in the `running` state for longer than the project's time limit do not time out.
- Pipelines cannot be started, retried or canceled. No new jobs can be created either.
2022-08-27 11:52:29 +05:30
- The status of the runners in `/admin/runners` won't be updated.
- `gitlab-runner verify` will return the error `ERROR: Verifying runner... is removed`.
2021-03-11 19:13:27 +05:30
After Maintenance Mode is disabled, new jobs are picked up again. Jobs that were
in the `running` state before enabling Maintenance Mode resume and their logs start
updating again.
NOTE:
It is recommended that you restart previously `running` pipelines after Maintenance Mode
is turned off.
### Deployments
2021-09-04 01:27:46 +05:30
Deployments don't go through because pipelines are unfinished.
2021-03-11 19:13:27 +05:30
2023-01-13 00:05:48 +05:30
It is recommended to disable auto deploys during Maintenance Mode, and enable them when it is disabled.
2021-03-11 19:13:27 +05:30
#### Terraform integration
2021-09-04 01:27:46 +05:30
Terraform integration depends on running CI pipelines, hence it is blocked.
2021-03-11 19:13:27 +05:30
### Container Registry
2021-09-04 01:27:46 +05:30
`docker push` fails with this error: `denied: requested access to the resource is denied`, but `docker pull` works.
2021-03-11 19:13:27 +05:30
### Package Registry
2021-09-04 01:27:46 +05:30
Package Registry allows you to install but not publish packages.
2021-03-11 19:13:27 +05:30
### Background jobs
2021-09-04 01:27:46 +05:30
Background jobs (cron jobs, Sidekiq) continue running as is, because background jobs are not automatically disabled.
2021-03-11 19:13:27 +05:30
2022-07-16 23:28:13 +05:30
[During a planned Geo failover](../geo/disaster_recovery/planned_failover.md#prevent-updates-to-the-primary-site),
2021-03-11 19:13:27 +05:30
it is recommended that you disable all cron jobs except for those related to Geo.
2021-09-04 01:27:46 +05:30
To monitor queues and disable jobs:
2022-10-11 01:57:18 +05:30
1. On the top bar, select **Main menu > Admin**.
2021-09-04 01:27:46 +05:30
1. On the left sidebar, select **Monitoring > Background Jobs**.
2021-03-11 19:13:27 +05:30
### Incident management
2021-09-04 01:27:46 +05:30
[Incident management](../../operations/incident_management/index.md) functions are limited. The creation of [alerts](../../operations/incident_management/alerts.md) and [incidents](../../operations/incident_management/incidents.md#incident-creation) are paused entirely. Notifications and paging on alerts and incidents are therefore disabled.
2021-03-11 19:13:27 +05:30
2021-04-17 20:07:23 +05:30
### Feature flags
2021-03-11 19:13:27 +05:30
- [Development feature flags](../../development/feature_flags/index.md) cannot be turned on or off through the API, but can be toggled through the Rails console.
2021-09-04 01:27:46 +05:30
- [The feature flag service](../../operations/feature_flags.md) responds to feature flag checks but feature flags cannot be toggled
2021-03-11 19:13:27 +05:30
### Geo secondaries
2021-09-04 01:27:46 +05:30
When primary is in Maintenance Mode, secondary also automatically goes into Maintenance Mode.
2021-03-11 19:13:27 +05:30
It is important that you do not disable replication before enabling Maintenance Mode.
2021-09-04 01:27:46 +05:30
Replication and verification continues to work but proxied Git pushes to primary do not work.
2021-03-11 19:13:27 +05:30
### Secure features
2022-03-02 08:16:31 +05:30
Features that depend on creating issues or creating or approving merge requests,
do not work.
2021-03-11 19:13:27 +05:30
2021-09-04 01:27:46 +05:30
Exporting a vulnerability list from a Vulnerability Report page does not work.
2021-03-11 19:13:27 +05:30
2021-09-04 01:27:46 +05:30
Changing the status on a finding or vulnerability object does not work, even though no error is shown in the UI.
2021-03-11 19:13:27 +05:30
SAST and Secret Detection cannot be initiated because they depend on passing CI jobs to create artifacts.
## An example use case: a planned failover
2021-09-04 01:27:46 +05:30
In the use case of [a planned failover](../geo/disaster_recovery/planned_failover.md), a few writes in the primary database are acceptable, since they are replicated quickly and are not significant in number.
2021-03-11 19:13:27 +05:30
For the same reason we don't automatically block background jobs when Maintenance Mode is enabled.
The resulting database writes are acceptable. Here, the trade-off is between more service degradation and the completion of replication.
2022-07-16 23:28:13 +05:30
However, during a planned failover, we [ask users to turn off cron jobs that are not related to Geo, manually](../geo/disaster_recovery/planned_failover.md#prevent-updates-to-the-primary-site). In the absence of new database writes and non-Geo cron jobs, new background jobs would either not be created at all or be minimal.