debian-mirror-gitlab/doc/operations/incident_management/manage_incidents.md

264 lines
10 KiB
Markdown
Raw Normal View History

2023-03-04 22:38:38 +05:30
---
stage: Monitor
group: Respond
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Manage incidents **(FREE)**
This page collects instructions for all the things you can do with [incidents](incidents.md) or in relation to them.
## Create an incident
You can create an incident manually or automatically.
### From the incidents list
> - [Moved](https://gitlab.com/gitlab-org/monitor/monitor/-/issues/24) to GitLab Free in 13.3.
> - [Permission changed](https://gitlab.com/gitlab-org/gitlab/-/issues/336624) from Guest to Reporter in GitLab 14.5.
> - Automatic application of the `incident` label [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/290964) in GitLab 14.8.
Prerequisites:
- You must have at least the Reporter role for the project.
To create an incident from the incidents list:
1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Monitor > Incidents**.
1. Select **Create incident**.
### From the issues list
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230857) in GitLab 13.4.
Prerequisites:
- You must have at least the Reporter role for the project.
To create an incident from the issues list:
1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Issues > List**, and select **New issue**.
1. From the **Type** dropdown list, select **Incident**. Only fields relevant to
incidents are available on the page.
1. Select **Create issue**.
### From an alert
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217745) in GitLab 13.1.
Create an incident issue when viewing an [alert](alerts.md).
The incident description is populated from the alert.
Prerequisites:
- You must have at least the Developer role for the project.
To create an incident from an alert:
1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Monitor > Alerts**.
1. Select your desired alert.
1. Select **Create incident**.
After an incident is created, to view it from the alert, select **View incident**.
When you [close an incident](#close-an-incident) linked to an alert, GitLab
[changes the alert's status](alerts.md#change-an-alerts-status) to **Resolved**.
You are then credited with the alert's status change.
### Automatically, when an alert is triggered **(ULTIMATE)**
In the project settings, you can turn on [creating an incident automatically](../metrics/alerts.md#trigger-actions-from-alerts)
whenever an alert is triggered.
### Using the PagerDuty webhook
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/119018) in GitLab 13.3.
> - [PagerDuty V3 Webhook](https://support.pagerduty.com/docs/webhooks) support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383029) in GitLab 15.7.
You can set up a webhook with PagerDuty to automatically create a GitLab incident
for each PagerDuty incident. This configuration requires you to make changes
in both PagerDuty and GitLab.
Prerequisites:
- You must have at least the Maintainer role for the project.
To set up a webhook with PagerDuty:
1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Settings > Monitor**
1. Expand **Incidents**.
1. Select the **PagerDuty integration** tab.
1. Turn on the **Active** toggle.
1. Select **Save integration**.
1. Copy the value of **Webhook URL** for use in a later step.
1. To add the webhook URL to a PagerDuty webhook integration, follow the steps described in the [PagerDuty documentation](https://support.pagerduty.com/docs/webhooks#manage-v3-webhook-subscriptions).
To confirm the integration is successful, trigger a test incident from PagerDuty to
check if a GitLab incident is created from the incident.
## View incidents list
To view the [incidents list](incidents.md#incidents-list):
1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Monitor > Incidents**.
To view an incident's [details page](incidents.md#incident-details), select it from the list.
### Who can view an incident
Whether you can view an incident depends on the [project visibility level](../../user/public_access.md) and
the incident's confidentiality status:
- Public project and a non-confidential incident: You don't have to be a member of the project.
- Private project and non-confidential incident: You must have at least the Guest role for the project.
- Confidential incident (regardless of project visibility): You must have at least the Reporter role for the project.
## Assign to a user
Assign incidents to users that are actively responding.
Prerequisites:
- You must have at least the Reporter role for the project.
To assign a user:
1. In an incident, on the right sidebar, next to **Assignees**, select **Edit**.
1. From the dropdown list, select one or [multiple users](../../user/project/issues/multiple_assignees_for_issues.md) to add as **assignees**.
1. Select any area outside the dropdown list.
## Change severity
> Editing severity on incident details page was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229402) in GitLab 13.4.
See [incident list](incidents.md#incidents-list) for a full description of the severity levels available.
Prerequisites:
- You must have at least the Reporter role for the project.
To change an incident's severity:
1. In an incident, on the right sidebar, next to **Severity**, select **Edit**.
1. From the dropdown list, select the new severity.
You can also change the severity using the `/severity` [quick action](../../user/project/quick_actions.md).
## Change status
> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5716) in GitLab 14.9 [with a flag](../../administration/feature_flags.md) named `incident_escalations`. Disabled by default.
> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) in GitLab 14.10.
> - [Feature flag `incident_escalations`](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) removed in GitLab 15.1.
Prerequisites:
- You must have at least the Developer role for the project.
To change the status of an incident:
1. In an incident, on the right sidebar, next to **Status**, select **Edit**.
1. From the dropdown list, select the new severity.
**Triggered** is the default status for new incidents.
### As an on-call responder **(PREMIUM)**
On-call responders can respond to [incident pages](paging.md#escalating-an-incident)
by changing the status.
Changing the status has the following effects:
- To **Acknowledged**: limits on-call pages based on the project's [escalation policy](escalation_policies.md).
- To **Resolved**: silences all on-call pages for the incident.
- From **Resolved** to **Triggered**: restarts the incident escalating.
In GitLab 15.1 and earlier, changing the status of an [incident created from an alert](#from-an-alert)
also changes the alert status. In [GitLab 15.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/356057),
the alert status is independent and does not change when the incident status changes.
## Change escalation policy **(PREMIUM)**
Prerequisites:
- You must have at least the Developer role for the project.
To change the escalation policy of an incident:
1. In an incident, on the right sidebar, next to **Escalation policy**, select **Edit**.
1. From the dropdown list, select the escalation policy.
By default, new incidents do not have an escalation policy selected.
Selecting an escalation policy [changes the incident status](#change-status) to **Triggered** and begins
[escalating the incident to on-call responders](paging.md#escalating-an-incident).
In GitLab 15.1 and earlier, the escalation policy for [incidents created from alerts](#from-an-alert)
reflects the alert's escalation policy and cannot be changed. In [GitLab 15.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/356057),
the incident escalation policy is independent and can be changed.
## Embed metrics
You can embed metrics anywhere [GitLab Flavored Markdown](../../user/markdown.md) is
used, like descriptions or comments. Embedding
metrics helps you share them when discussing incidents or performance issues.
To embed metrics in a Markdown text box in GitLab,
[paste the link to the dashboard](../metrics/embed.md#embedding-gitlab-managed-kubernetes-metrics).
You can embed both [GitLab-hosted metrics](../metrics/embed.md) (deprecated) and
[Grafana metrics](../metrics/embed_grafana.md) in incidents and issue
templates.
## Close an incident
Prerequisites:
- You must have at least the Reporter role for the project.
To close an incident, in the top right, select **Close incident**.
When you close an incident that is linked to an [alert](alerts.md),
the linked alert's status changes to **Resolved**.
You are then credited with the alert's status change.
### Automatically close incidents via recovery alerts
> [Introduced for HTTP integrations](https://gitlab.com/gitlab-org/gitlab/-/issues/13402) in GitLab 13.4.
Turn on closing an incident automatically when GitLab receives a recovery alert
from a HTTP or Prometheus webhook.
Prerequisites:
- You must have at least the Maintainer role for the project.
To configure the setting:
1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Settings > Monitor**.
1. Expand the **Incidents** section.
1. Select the **Automatically close associated incident** checkbox.
1. Select **Save changes**.
When GitLab receives a recovery alert, it closes the associated incident.
This action is recorded as a system note on the incident indicating that it
was closed automatically by the GitLab Alert bot.
## Other actions
Because incidents in GitLab are built on top of [issues](../../user/project/issues/index.md),
they have the following actions in common:
- [Add a to-do item](../../user/todos.md#create-a-to-do-item)
- [Add labels](../../user/project/labels.md#assign-and-unassign-labels)
- [Assign a milestone](../../user/project/milestones/index.md#assign-a-milestone-to-an-issue-or-merge-request)
- [Make an incident confidential](../../user/project/issues/confidential_issues.md)
- [Set a due date](../../user/project/issues/due_dates.md)
- [Toggle notifications](../../user/profile/notifications.md#edit-notification-settings-for-issues-merge-requests-and-epics)
- [Track time spent](../../user/project/time_tracking.md)