2020-07-28 23:09:34 +05:30
---
stage: Monitor
2022-04-04 11:22:00 +05:30
group: Respond
2022-11-25 23:54:43 +05:30
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
2020-07-28 23:09:34 +05:30
---
2021-03-11 19:13:27 +05:30
# Set up alerts for Prometheus metrics **(FREE)**
2020-07-28 23:09:34 +05:30
After [configuring metrics for your CI/CD environment ](index.md ), you can set up
2021-09-30 23:02:18 +05:30
alerting for Prometheus metrics, and
2020-11-24 15:15:51 +05:30
[trigger actions from alerts ](#trigger-actions-from-alerts ) to notify
2020-07-28 23:09:34 +05:30
your team when environment performance falls outside of the boundaries you set.
2021-06-08 01:23:25 +05:30
## Prometheus cluster integrations
2023-03-04 22:38:38 +05:30
Alerts are not supported for [Prometheus cluster integrations ](../../user/clusters/integrations.md ).
2021-06-08 01:23:25 +05:30
2020-07-28 23:09:34 +05:30
## Trigger actions from alerts **(ULTIMATE)**
2023-03-04 22:38:38 +05:30
> - Introduced in GitLab 13.1: incidents are not created automatically by default .
> - Mapping common severity values from the alert payload ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50871) in GitLab 13.9.
Turn on creating [incidents ](../incident_management/incidents.md ) automatically whenever an alert is triggered.
Prerequisites:
- You must have at least the Maintainer role for the project.
2020-07-28 23:09:34 +05:30
2023-03-04 22:38:38 +05:30
To configure the actions:
1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Settings > Monitor** .
1. Expand the **Alerts** section, then select the **Alert settings** tab.
1. Select the **Create an incident** checkbox.
1. Optional. To customize the incident, from the **Incident template** , select a template to be
appended to the [incident summary ](../incident_management/incidents.md#summary ).
If the dropdown list is empty,
[create an issue template ](../../user/project/description_templates.md#create-an-issue-template ) first.
1. Optional. To send [an email notification ](../incident_management/paging.md#email-notifications-for-alerts ), select the
**Send a single email notification to Owners and Maintainers for new alerts** checkbox.
2022-07-23 23:45:48 +05:30
1. Select **Save changes** .
2020-07-28 23:09:34 +05:30
2023-03-04 22:38:38 +05:30
### Fields in automatically created incidents
Incidents [created automatically from an alert ](#trigger-actions-from-alerts ) are filled with
values extracted from the `alerts` field in the
[webhook payload ](https://prometheus.io/docs/alerting/latest/configuration/#webhook_config ):
2020-07-28 23:09:34 +05:30
2023-03-04 22:38:38 +05:30
- Incident author: `GitLab Alert Bot`
- Incident title: Extracted from the alert payload fields `annotations/title` , `annotations/summary` , or `labels/alertname` .
- Incident description: Extracted from alert payload field `annotations/description` .
2020-10-24 23:57:45 +05:30
- Alert `Summary` : A list of properties from the alert's payload.
- `starts_at` : Alert start time from the payload's `startsAt` field
- `full_query` : Alert query extracted from the payload's `generatorURL` field
2020-07-28 23:09:34 +05:30
- Optional list of attached annotations extracted from `annotations/*`
2022-06-21 17:19:12 +05:30
- Alert [GLFM ](../../user/markdown.md ): GitLab Flavored Markdown from the payload's `annotations/gitlab_incident_markdown` field.
2023-03-04 22:38:38 +05:30
- Alert severity:
2021-03-11 19:13:27 +05:30
Extracted from the alert payload field `labels/severity` . Maps case-insensitive
value to [Alert's severity ](../incident_management/alerts.md#alert-severity ):
2023-03-17 16:20:25 +05:30
| Alert payload | Mapped to alert severity |
| ------------- | --------------------------------------------------------------------------- |
| Critical | `critical` , `s1` , `p1` , `emergency` , `fatal` , or any value not in this list |
| High | `high` , `s2` , `p2` , `major` , `page` |
| Medium | `medium` , `s3` , `p3` , `error` , `alert` |
| Low | `low` , `s4` , `p4` , `warn` , `warning` |
| Info | `info` , `s5` , `p5` , `debug` , `information` , `notice` |
2020-07-28 23:09:34 +05:30
2023-03-04 22:38:38 +05:30
To further customize the incident, you can add labels, mentions, or any other supported
2020-07-28 23:09:34 +05:30
[quick action ](../../user/project/quick_actions.md ) in the selected issue template,
which applies to all incidents. To limit quick actions or other information to
only specific types of alerts, use the `annotations/gitlab_incident_markdown` field.
GitLab tags each incident issue with the `incident` label automatically. If the label
2023-03-04 22:38:38 +05:30
does not yet exist, it's created automatically.
2021-06-08 01:23:25 +05:30
### Recovery alerts
2023-03-04 22:38:38 +05:30
The alert in GitLab is automatically resolved when Prometheus
2021-06-08 01:23:25 +05:30
sends a payload with the field `status` set to `resolved` .
2023-03-04 22:38:38 +05:30
You can also configure the associated [incident to be closed automatically ](../incident_management/manage_incidents.md#automatically-close-incidents-via-recovery-alerts ) when the alert resolves.
