130 lines
6.5 KiB
Markdown
130 lines
6.5 KiB
Markdown
---
|
|
stage: Monitor
|
|
group: Respond
|
|
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
|
|
---
|
|
|
|
# Prometheus **(FREE)**
|
|
|
|
GitLab offers powerful integration with [Prometheus](https://prometheus.io) for
|
|
monitoring key metrics of your apps, directly in GitLab.
|
|
Metrics for each environment are retrieved from Prometheus, and then displayed
|
|
in the GitLab interface.
|
|
|
|
|
|
|
|
There are two ways to set up Prometheus integration, depending on where your apps are running:
|
|
|
|
- For deployments on Kubernetes, GitLab can be [integrated with an in-cluster Prometheus](#prometheus-cluster-integration)
|
|
- For other deployment targets, [specify the Prometheus server](#manual-configuration-of-prometheus).
|
|
|
|
Once enabled, GitLab detects metrics from known services in the
|
|
[metric library](prometheus_library/index.md). You can also
|
|
[add your own metrics](../../../operations/metrics/index.md#adding-custom-metrics) and create
|
|
[custom dashboards](../../../operations/metrics/dashboards/index.md).
|
|
|
|
## Enabling the Prometheus integration
|
|
|
|
### Prometheus cluster integration
|
|
|
|
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/55244) in GitLab 13.11.
|
|
> - [Replaced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62725) the Prometheus cluster applications in GitLab 14.0.
|
|
|
|
GitLab can query an in-cluster Prometheus for your metrics.
|
|
See [Prometheus cluster integration](../../clusters/integrations.md#prometheus-cluster-integration) for details.
|
|
|
|
### Manual configuration of Prometheus
|
|
|
|
#### Requirements
|
|
|
|
Integration with Prometheus requires the following:
|
|
|
|
- Prometheus must be configured to collect one of the [supported metrics](prometheus_library/index.md)
|
|
- Each metric must have a label to indicate the environment
|
|
- GitLab must have network connectivity to the Prometheus server
|
|
|
|
#### Getting started
|
|
|
|
Installing and configuring Prometheus to monitor applications is fairly straightforward.
|
|
|
|
1. [Install Prometheus](https://prometheus.io/docs/prometheus/latest/installation/)
|
|
1. Set up one of the [supported monitoring targets](prometheus_library/index.md)
|
|
1. Configure the Prometheus server to
|
|
[collect their metrics](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config)
|
|
|
|
#### Configuration in GitLab
|
|
|
|
The actual configuration of Prometheus integration in GitLab
|
|
requires the domain name or IP address of the Prometheus server you'd like
|
|
to integrate with. If the Prometheus resource is secured with Google's Identity-Aware Proxy (IAP),
|
|
you can pass information like Client ID and Service Account credentials.
|
|
GitLab can use these to access the resource. More information about authentication from a
|
|
service account can be found at Google's documentation for
|
|
[Authenticating from a service account](https://cloud.google.com/iap/docs/authentication-howto#authenticating_from_a_service_account).
|
|
|
|
1. On the top bar, select **Main menu > Projects** and find your project.
|
|
1. On the left sidebar, select **Settings > Integrations**.
|
|
1. Select **Prometheus**.
|
|
1. For **API URL**, provide the domain name or IP address of your server, such as
|
|
`http://prometheus.example.com/` or `http://192.0.2.1/`.
|
|
1. (Optional) In **Google IAP Audience Client ID**, provide the Client ID of the
|
|
Prometheus OAuth Client secured with Google IAP.
|
|
1. (Optional) In **Google IAP Service Account JSON**, provide the contents of the
|
|
Service Account credentials file that is authorized to access the Prometheus resource.
|
|
The JSON key `token_credential_uri` is discarded to prevent
|
|
[Server-side Request Forgery (SSRF)](https://www.hackerone.com/application-security/how-server-side-request-forgery-ssrf).
|
|
1. Select **Save changes**.
|
|
|
|
|
|
|
|
#### Thanos configuration in GitLab
|
|
|
|
You can configure [Thanos](https://thanos.io/) as a drop-in replacement for Prometheus
|
|
with GitLab. Use the domain name or IP address of the Thanos server you'd like
|
|
to integrate with.
|
|
|
|
1. On the top bar, select **Main menu > Projects** and find your project.
|
|
1. On the left sidebar, select **Settings > Integrations**.
|
|
1. Select **Prometheus**.
|
|
1. Provide the domain name or IP address of your server, for example
|
|
`http://thanos.example.com/` or `http://192.0.2.1/`.
|
|
1. Select **Save changes**.
|
|
|
|
### Precedence with multiple Prometheus configurations
|
|
|
|
Although you can enable both a [manual configuration](#manual-configuration-of-prometheus)
|
|
and [cluster integration](#prometheus-cluster-integration) of Prometheus, you
|
|
can use only one:
|
|
|
|
- If you have enabled a
|
|
[Prometheus manual configuration](#manual-configuration-of-prometheus)
|
|
and a [Prometheus cluster integration](#prometheus-cluster-integration),
|
|
the manual configuration takes precedence and is used to run queries from
|
|
[custom dashboards](../../../operations/metrics/dashboards/index.md) and
|
|
[custom metrics](../../../operations/metrics/index.md#adding-custom-metrics).
|
|
- If you have managed Prometheus applications installed on Kubernetes clusters
|
|
at **different** levels (project, group, instance), the order of precedence is described in
|
|
[Cluster precedence](../../instance/clusters/index.md#cluster-precedence).
|
|
- If you have managed Prometheus applications installed on multiple Kubernetes
|
|
clusters at the **same** level, the Prometheus application of a cluster with a
|
|
matching [environment scope](../../../ci/environments/index.md#limit-the-environment-scope-of-a-cicd-variable) is used.
|
|
|
|
## Determining the performance impact of a merge
|
|
|
|
Developers can view the performance impact of their changes in the merge
|
|
request workflow. This feature requires [Kubernetes](prometheus_library/kubernetes.md) metrics.
|
|
|
|
When a source branch has been deployed to an environment, a sparkline and
|
|
numeric comparison of the average memory consumption displays. On the
|
|
sparkline, a dot indicates when the current changes were deployed, with up to 30 minutes of
|
|
performance data displayed before and after. The comparison shows the difference
|
|
between the 30 minute average before and after the deployment. This information
|
|
is updated after each commit has been deployed.
|
|
|
|
Once merged and the target branch has been redeployed, the metrics switches
|
|
to show the new environments this revision has been deployed to.
|
|
|
|
Performance data is available for the duration it is persisted on the
|
|
Prometheus server.
|
|
|
|
|