263 lines
10 KiB
Markdown
263 lines
10 KiB
Markdown
---
|
|
stage: Monitor
|
|
group: Respond
|
|
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
|
|
---
|
|
|
|
# Manage incidents **(FREE)**
|
|
|
|
This page collects instructions for all the things you can do with [incidents](incidents.md) or in relation to them.
|
|
|
|
## Create an incident
|
|
|
|
You can create an incident manually or automatically.
|
|
|
|
### From the incidents list
|
|
|
|
> - [Moved](https://gitlab.com/gitlab-org/monitor/monitor/-/issues/24) to GitLab Free in 13.3.
|
|
> - [Permission changed](https://gitlab.com/gitlab-org/gitlab/-/issues/336624) from Guest to Reporter in GitLab 14.5.
|
|
> - Automatic application of the `incident` label [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/290964) in GitLab 14.8.
|
|
|
|
Prerequisites:
|
|
|
|
- You must have at least the Reporter role for the project.
|
|
|
|
To create an incident from the incidents list:
|
|
|
|
1. On the top bar, select **Main menu > Projects** and find your project.
|
|
1. On the left sidebar, select **Monitor > Incidents**.
|
|
1. Select **Create incident**.
|
|
|
|
### From the issues list
|
|
|
|
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230857) in GitLab 13.4.
|
|
|
|
Prerequisites:
|
|
|
|
- You must have at least the Reporter role for the project.
|
|
|
|
To create an incident from the issues list:
|
|
|
|
1. On the top bar, select **Main menu > Projects** and find your project.
|
|
1. On the left sidebar, select **Issues > List**, and select **New issue**.
|
|
1. From the **Type** dropdown list, select **Incident**. Only fields relevant to
|
|
incidents are available on the page.
|
|
1. Select **Create issue**.
|
|
|
|
### From an alert
|
|
|
|
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217745) in GitLab 13.1.
|
|
|
|
Create an incident issue when viewing an [alert](alerts.md).
|
|
The incident description is populated from the alert.
|
|
|
|
Prerequisites:
|
|
|
|
- You must have at least the Developer role for the project.
|
|
|
|
To create an incident from an alert:
|
|
|
|
1. On the top bar, select **Main menu > Projects** and find your project.
|
|
1. On the left sidebar, select **Monitor > Alerts**.
|
|
1. Select your desired alert.
|
|
1. Select **Create incident**.
|
|
|
|
After an incident is created, to view it from the alert, select **View incident**.
|
|
|
|
When you [close an incident](#close-an-incident) linked to an alert, GitLab
|
|
[changes the alert's status](alerts.md#change-an-alerts-status) to **Resolved**.
|
|
You are then credited with the alert's status change.
|
|
|
|
### Automatically, when an alert is triggered **(ULTIMATE)**
|
|
|
|
In the project settings, you can turn on [creating an incident automatically](../metrics/alerts.md#trigger-actions-from-alerts)
|
|
whenever an alert is triggered.
|
|
|
|
### Using the PagerDuty webhook
|
|
|
|
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/119018) in GitLab 13.3.
|
|
> - [PagerDuty V3 Webhook](https://support.pagerduty.com/docs/webhooks) support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383029) in GitLab 15.7.
|
|
|
|
You can set up a webhook with PagerDuty to automatically create a GitLab incident
|
|
for each PagerDuty incident. This configuration requires you to make changes
|
|
in both PagerDuty and GitLab.
|
|
|
|
Prerequisites:
|
|
|
|
- You must have at least the Maintainer role for the project.
|
|
|
|
To set up a webhook with PagerDuty:
|
|
|
|
1. On the top bar, select **Main menu > Projects** and find your project.
|
|
1. On the left sidebar, select **Settings > Monitor**
|
|
1. Expand **Incidents**.
|
|
1. Select the **PagerDuty integration** tab.
|
|
1. Turn on the **Active** toggle.
|
|
1. Select **Save integration**.
|
|
1. Copy the value of **Webhook URL** for use in a later step.
|
|
1. To add the webhook URL to a PagerDuty webhook integration, follow the steps described in the [PagerDuty documentation](https://support.pagerduty.com/docs/webhooks#manage-v3-webhook-subscriptions).
|
|
|
|
To confirm the integration is successful, trigger a test incident from PagerDuty to
|
|
check if a GitLab incident is created from the incident.
|
|
|
|
## View incidents list
|
|
|
|
To view the [incidents list](incidents.md#incidents-list):
|
|
|
|
1. On the top bar, select **Main menu > Projects** and find your project.
|
|
1. On the left sidebar, select **Monitor > Incidents**.
|
|
|
|
To view an incident's [details page](incidents.md#incident-details), select it from the list.
|
|
|
|
### Who can view an incident
|
|
|
|
Whether you can view an incident depends on the [project visibility level](../../user/public_access.md) and
|
|
the incident's confidentiality status:
|
|
|
|
- Public project and a non-confidential incident: You don't have to be a member of the project.
|
|
- Private project and non-confidential incident: You must have at least the Guest role for the project.
|
|
- Confidential incident (regardless of project visibility): You must have at least the Reporter role for the project.
|
|
|
|
## Assign to a user
|
|
|
|
Assign incidents to users that are actively responding.
|
|
|
|
Prerequisites:
|
|
|
|
- You must have at least the Reporter role for the project.
|
|
|
|
To assign a user:
|
|
|
|
1. In an incident, on the right sidebar, next to **Assignees**, select **Edit**.
|
|
1. From the dropdown list, select one or [multiple users](../../user/project/issues/multiple_assignees_for_issues.md) to add as **assignees**.
|
|
1. Select any area outside the dropdown list.
|
|
|
|
## Change severity
|
|
|
|
> Editing severity on incident details page was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229402) in GitLab 13.4.
|
|
|
|
See [incident list](incidents.md#incidents-list) for a full description of the severity levels available.
|
|
|
|
Prerequisites:
|
|
|
|
- You must have at least the Reporter role for the project.
|
|
|
|
To change an incident's severity:
|
|
|
|
1. In an incident, on the right sidebar, next to **Severity**, select **Edit**.
|
|
1. From the dropdown list, select the new severity.
|
|
|
|
You can also change the severity using the `/severity` [quick action](../../user/project/quick_actions.md).
|
|
|
|
## Change status
|
|
|
|
> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5716) in GitLab 14.9 [with a flag](../../administration/feature_flags.md) named `incident_escalations`. Disabled by default.
|
|
> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) in GitLab 14.10.
|
|
> - [Feature flag `incident_escalations`](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) removed in GitLab 15.1.
|
|
|
|
Prerequisites:
|
|
|
|
- You must have at least the Developer role for the project.
|
|
|
|
To change the status of an incident:
|
|
|
|
1. In an incident, on the right sidebar, next to **Status**, select **Edit**.
|
|
1. From the dropdown list, select the new severity.
|
|
|
|
**Triggered** is the default status for new incidents.
|
|
|
|
### As an on-call responder **(PREMIUM)**
|
|
|
|
On-call responders can respond to [incident pages](paging.md#escalating-an-incident)
|
|
by changing the status.
|
|
|
|
Changing the status has the following effects:
|
|
|
|
- To **Acknowledged**: limits on-call pages based on the project's [escalation policy](escalation_policies.md).
|
|
- To **Resolved**: silences all on-call pages for the incident.
|
|
- From **Resolved** to **Triggered**: restarts the incident escalating.
|
|
|
|
In GitLab 15.1 and earlier, changing the status of an [incident created from an alert](#from-an-alert)
|
|
also changes the alert status. In [GitLab 15.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/356057),
|
|
the alert status is independent and does not change when the incident status changes.
|
|
|
|
## Change escalation policy **(PREMIUM)**
|
|
|
|
Prerequisites:
|
|
|
|
- You must have at least the Developer role for the project.
|
|
|
|
To change the escalation policy of an incident:
|
|
|
|
1. In an incident, on the right sidebar, next to **Escalation policy**, select **Edit**.
|
|
1. From the dropdown list, select the escalation policy.
|
|
|
|
By default, new incidents do not have an escalation policy selected.
|
|
|
|
Selecting an escalation policy [changes the incident status](#change-status) to **Triggered** and begins
|
|
[escalating the incident to on-call responders](paging.md#escalating-an-incident).
|
|
|
|
In GitLab 15.1 and earlier, the escalation policy for [incidents created from alerts](#from-an-alert)
|
|
reflects the alert's escalation policy and cannot be changed. In [GitLab 15.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/356057),
|
|
the incident escalation policy is independent and can be changed.
|
|
|
|
## Embed metrics
|
|
|
|
You can embed metrics anywhere [GitLab Flavored Markdown](../../user/markdown.md) is
|
|
used, like descriptions or comments. Embedding
|
|
metrics helps you share them when discussing incidents or performance issues.
|
|
|
|
To embed metrics in a Markdown text box in GitLab,
|
|
[paste the link to the dashboard](../metrics/embed.md#embedding-gitlab-managed-kubernetes-metrics).
|
|
|
|
You can embed both [GitLab-hosted metrics](../metrics/embed.md) (deprecated) and
|
|
[Grafana metrics](../metrics/embed_grafana.md) in incidents and issue
|
|
templates.
|
|
|
|
## Close an incident
|
|
|
|
Prerequisites:
|
|
|
|
- You must have at least the Reporter role for the project.
|
|
|
|
To close an incident, in the top right, select **Close incident**.
|
|
|
|
When you close an incident that is linked to an [alert](alerts.md),
|
|
the linked alert's status changes to **Resolved**.
|
|
You are then credited with the alert's status change.
|
|
|
|
### Automatically close incidents via recovery alerts
|
|
|
|
> [Introduced for HTTP integrations](https://gitlab.com/gitlab-org/gitlab/-/issues/13402) in GitLab 13.4.
|
|
|
|
Turn on closing an incident automatically when GitLab receives a recovery alert
|
|
from a HTTP or Prometheus webhook.
|
|
|
|
Prerequisites:
|
|
|
|
- You must have at least the Maintainer role for the project.
|
|
|
|
To configure the setting:
|
|
|
|
1. On the top bar, select **Main menu > Projects** and find your project.
|
|
1. On the left sidebar, select **Settings > Monitor**.
|
|
1. Expand the **Incidents** section.
|
|
1. Select the **Automatically close associated incident** checkbox.
|
|
1. Select **Save changes**.
|
|
|
|
When GitLab receives a recovery alert, it closes the associated incident.
|
|
This action is recorded as a system note on the incident indicating that it
|
|
was closed automatically by the GitLab Alert bot.
|
|
|
|
## Other actions
|
|
|
|
Because incidents in GitLab are built on top of [issues](../../user/project/issues/index.md),
|
|
they have the following actions in common:
|
|
|
|
- [Add a to-do item](../../user/todos.md#create-a-to-do-item)
|
|
- [Add labels](../../user/project/labels.md#assign-and-unassign-labels)
|
|
- [Assign a milestone](../../user/project/milestones/index.md#assign-a-milestone-to-an-issue-or-merge-request)
|
|
- [Make an incident confidential](../../user/project/issues/confidential_issues.md)
|
|
- [Set a due date](../../user/project/issues/due_dates.md)
|
|
- [Toggle notifications](../../user/profile/notifications.md#edit-notification-settings-for-issues-merge-requests-and-epics)
|
|
- [Track time spent](../../user/project/time_tracking.md)
|