debian-mirror-gitlab/doc/operations/incident_management/manage_incidents.md

stage	group	info
Monitor	Respond	To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments

Manage incidents (FREE)

This page collects instructions for all the things you can do with incidents or in relation to them.

Create an incident

You can create an incident manually or automatically.

From the incidents list

• Moved to GitLab Free in 13.3.
• Permission changed from Guest to Reporter in GitLab 14.5.
• Automatic application of the incident label removed in GitLab 14.8.

Prerequisites:

• You must have at least the Reporter role for the project.

To create an incident from the incidents list:

1. On the top bar, select Main menu > Projects and find your project.
2. On the left sidebar, select Monitor > Incidents.
3. Select Create incident.

From the issues list

Introduced in GitLab 13.4.

Prerequisites:

• You must have at least the Reporter role for the project.

To create an incident from the issues list:

1. On the top bar, select Main menu > Projects and find your project.
2. On the left sidebar, select Issues > List, and select New issue.
3. From the Type dropdown list, select Incident. Only fields relevant to incidents are available on the page.
4. Select Create issue.

From an alert

Introduced in GitLab 13.1.

Create an incident issue when viewing an alert. The incident description is populated from the alert.

Prerequisites:

• You must have at least the Developer role for the project.

To create an incident from an alert:

1. On the top bar, select Main menu > Projects and find your project.
2. On the left sidebar, select Monitor > Alerts.
3. Select your desired alert.
4. Select Create incident.

After an incident is created, to view it from the alert, select View incident.

When you close an incident linked to an alert, GitLab changes the alert's status to Resolved. You are then credited with the alert's status change.

Automatically, when an alert is triggered (ULTIMATE)

In the project settings, you can turn on creating an incident automatically whenever an alert is triggered.

Using the PagerDuty webhook

• Introduced in GitLab 13.3.
• PagerDuty V3 Webhook support introduced in GitLab 15.7.

You can set up a webhook with PagerDuty to automatically create a GitLab incident for each PagerDuty incident. This configuration requires you to make changes in both PagerDuty and GitLab.

Prerequisites:

• You must have at least the Maintainer role for the project.

To set up a webhook with PagerDuty:

1. On the top bar, select Main menu > Projects and find your project.
2. On the left sidebar, select Settings > Monitor
3. Expand Incidents.
4. Select the PagerDuty integration tab.
5. Turn on the Active toggle.
6. Select Save integration.
7. Copy the value of Webhook URL for use in a later step.
8. To add the webhook URL to a PagerDuty webhook integration, follow the steps described in the PagerDuty documentation.

To confirm the integration is successful, trigger a test incident from PagerDuty to check if a GitLab incident is created from the incident.

View incidents list

To view the incidents list:

1. On the top bar, select Main menu > Projects and find your project.
2. On the left sidebar, select Monitor > Incidents.

To view an incident's details page, select it from the list.

Who can view an incident

Whether you can view an incident depends on the project visibility level and the incident's confidentiality status:

• Public project and a non-confidential incident: You don't have to be a member of the project.
• Private project and non-confidential incident: You must have at least the Guest role for the project.
• Confidential incident (regardless of project visibility): You must have at least the Reporter role for the project.

Assign to a user

Assign incidents to users that are actively responding.

Prerequisites:

• You must have at least the Reporter role for the project.

To assign a user:

1. In an incident, on the right sidebar, next to Assignees, select Edit.
2. From the dropdown list, select one or multiple users to add as assignees.
3. Select any area outside the dropdown list.

Change severity

Editing severity on incident details page was introduced in GitLab 13.4.

See incident list for a full description of the severity levels available.

Prerequisites:

• You must have at least the Reporter role for the project.

To change an incident's severity:

1. In an incident, on the right sidebar, next to Severity, select Edit.
2. From the dropdown list, select the new severity.

You can also change the severity using the /severity quick action.

Change status

• Introduced in GitLab 14.9 with a flag named incident_escalations. Disabled by default.
• Enabled on GitLab.com and self-managed in GitLab 14.10.
• Feature flag incident_escalations removed in GitLab 15.1.

Prerequisites:

• You must have at least the Developer role for the project.

To change the status of an incident:

1. In an incident, on the right sidebar, next to Status, select Edit.
2. From the dropdown list, select the new severity.

Triggered is the default status for new incidents.

As an on-call responder (PREMIUM)

On-call responders can respond to incident pages by changing the status.

Changing the status has the following effects:

• To Acknowledged: limits on-call pages based on the project's escalation policy.
• To Resolved: silences all on-call pages for the incident.
• From Resolved to Triggered: restarts the incident escalating.

In GitLab 15.1 and earlier, changing the status of an incident created from an alert also changes the alert status. In GitLab 15.2 and later, the alert status is independent and does not change when the incident status changes.

Change escalation policy (PREMIUM)

Prerequisites:

• You must have at least the Developer role for the project.

To change the escalation policy of an incident:

1. In an incident, on the right sidebar, next to Escalation policy, select Edit.
2. From the dropdown list, select the escalation policy.

By default, new incidents do not have an escalation policy selected.

Selecting an escalation policy changes the incident status to Triggered and begins escalating the incident to on-call responders.

In GitLab 15.1 and earlier, the escalation policy for incidents created from alerts reflects the alert's escalation policy and cannot be changed. In GitLab 15.2 and later, the incident escalation policy is independent and can be changed.

Embed metrics

You can embed metrics anywhere GitLab Flavored Markdown is used, like descriptions or comments. Embedding metrics helps you share them when discussing incidents or performance issues.

To embed metrics in a Markdown text box in GitLab, paste the link to the dashboard.

You can embed both GitLab-hosted metrics (deprecated) and Grafana metrics in incidents and issue templates.

Close an incident

Prerequisites:

• You must have at least the Reporter role for the project.

To close an incident, in the upper right, select Close incident.

When you close an incident that is linked to an alert, the linked alert's status changes to Resolved. You are then credited with the alert's status change.

Automatically close incidents via recovery alerts

Introduced for HTTP integrations in GitLab 13.4.

Turn on closing an incident automatically when GitLab receives a recovery alert from a HTTP or Prometheus webhook.

Prerequisites:

• You must have at least the Maintainer role for the project.

To configure the setting:

1. On the top bar, select Main menu > Projects and find your project.
2. On the left sidebar, select Settings > Monitor.
3. Expand the Incidents section.
4. Select the Automatically close associated incident checkbox.
5. Select Save changes.

When GitLab receives a recovery alert, it closes the associated incident. This action is recorded as a system note on the incident indicating that it was closed automatically by the GitLab Alert bot.

Other actions

Because incidents in GitLab are built on top of issues, they have the following actions in common:

• Add a to-do item
• Add labels
• Assign a milestone
• Make an incident confidential
• Set a due date
• Toggle notifications
• Track time spent