debian-mirror-gitlab/doc/operations/metrics/alerts.md

81 lines
4.3 KiB
Markdown
Raw Normal View History

2020-07-28 23:09:34 +05:30
---
stage: Monitor
2022-04-04 11:22:00 +05:30
group: Respond
2021-02-22 17:27:13 +05:30
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
2020-07-28 23:09:34 +05:30
---
2021-03-11 19:13:27 +05:30
# Set up alerts for Prometheus metrics **(FREE)**
2020-07-28 23:09:34 +05:30
2021-03-11 19:13:27 +05:30
> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/42640) to GitLab Free in 12.10.
2020-11-24 15:15:51 +05:30
2020-07-28 23:09:34 +05:30
After [configuring metrics for your CI/CD environment](index.md), you can set up
2021-09-30 23:02:18 +05:30
alerting for Prometheus metrics, and
2020-11-24 15:15:51 +05:30
[trigger actions from alerts](#trigger-actions-from-alerts) to notify
2020-07-28 23:09:34 +05:30
your team when environment performance falls outside of the boundaries you set.
2021-06-08 01:23:25 +05:30
## Prometheus cluster integrations
Alerts are not currently supported for [Prometheus cluster integrations](../../user/clusters/integrations.md).
2022-07-23 23:45:48 +05:30
<!--- start_remove The following content will be removed on remove_date: '2022-09-22' -->
2020-07-28 23:09:34 +05:30
2022-07-23 23:45:48 +05:30
## External Prometheus instances (removed)
2020-07-28 23:09:34 +05:30
2022-07-23 23:45:48 +05:30
This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/219142) in GitLab 13.2 and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/338834) in 15.0.
To manually configure a Prometheus server, we recommend
you use the [generic alerts integration](../incident_management/integrations.md).
2021-03-11 19:13:27 +05:30
2022-07-23 23:45:48 +05:30
<!--- end_remove -->
2020-07-28 23:09:34 +05:30
## Trigger actions from alerts **(ULTIMATE)**
Alerts can be used to trigger actions, like opening an issue automatically
(disabled by default since `13.1`). To configure the actions:
2021-09-04 01:27:46 +05:30
1. Navigate to your project's **Settings > Monitor > Alerts**.
2020-07-28 23:09:34 +05:30
1. Enable the option to create issues.
1. Choose the [issue template](../../user/project/description_templates.md) to create the issue from.
1. Optionally, select whether to send an email notification to the developers of the project.
2022-07-23 23:45:48 +05:30
1. Select **Save changes**.
2020-07-28 23:09:34 +05:30
After enabling, GitLab automatically opens an issue when an alert is triggered containing
2020-10-24 23:57:45 +05:30
values extracted from the [`alerts` field in webhook payload](https://prometheus.io/docs/alerting/latest/configuration/#webhook_config):
2020-07-28 23:09:34 +05:30
- Issue author: `GitLab Alert Bot`
2020-10-24 23:57:45 +05:30
- Issue title: Extracted from the alert payload fields `annotations/title`, `annotations/summary`, or `labels/alertname`.
2021-03-11 19:13:27 +05:30
- Issue description: Extracted from alert payload field `annotations/description`.
2020-10-24 23:57:45 +05:30
- Alert `Summary`: A list of properties from the alert's payload.
- `starts_at`: Alert start time from the payload's `startsAt` field
- `full_query`: Alert query extracted from the payload's `generatorURL` field
2020-07-28 23:09:34 +05:30
- Optional list of attached annotations extracted from `annotations/*`
2022-06-21 17:19:12 +05:30
- Alert [GLFM](../../user/markdown.md): GitLab Flavored Markdown from the payload's `annotations/gitlab_incident_markdown` field.
2021-11-18 22:05:49 +05:30
- Alert Severity ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50871) in GitLab version 13.9):
2021-03-11 19:13:27 +05:30
Extracted from the alert payload field `labels/severity`. Maps case-insensitive
value to [Alert's severity](../incident_management/alerts.md#alert-severity):
- **Critical**: `critical`, `s1`, `p1`, `emergency`, `fatal`, or any value not in this list
- **High**: `high`, `s2`, `p2`, `major`, `page`
- **Medium**: `medium`, `s3`, `p3`, `error`, `alert`
- **Low**: `low`, `s4`, `p4`, `warn`, `warning`
- **Info**: `info`, `s5`, `p5`, `debug`, `information`, `notice`
2020-07-28 23:09:34 +05:30
To further customize the issue, you can add labels, mentions, or any other supported
[quick action](../../user/project/quick_actions.md) in the selected issue template,
which applies to all incidents. To limit quick actions or other information to
only specific types of alerts, use the `annotations/gitlab_incident_markdown` field.
Since [version 12.2](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/63373),
GitLab tags each incident issue with the `incident` label automatically. If the label
does not yet exist, it is also created automatically.
If the metric exceeds the threshold of the alert for over 5 minutes, GitLab sends
2022-04-04 11:22:00 +05:30
an email to all Maintainers and Owners of the project.
2021-06-08 01:23:25 +05:30
### Recovery alerts
2021-11-18 22:05:49 +05:30
> [From GitLab 12.5](https://gitlab.com/gitlab-org/gitlab/-/issues/13401), when GitLab receives a recovery alert, it automatically closes the associated issue.
2021-06-08 01:23:25 +05:30
The alert in GitLab will be automatically resolved when Prometheus
sends a payload with the field `status` set to `resolved`.
You can also configure the associated [incident to be closed automatically](../incident_management/incidents.md#automatically-close-incidents-via-recovery-alerts) when the alert resolves.