debian-mirror-gitlab/doc/development/service_ping/index.md

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

511 lines
18 KiB
Markdown
Raw Normal View History

2021-09-30 23:02:18 +05:30
---
2022-08-13 15:12:31 +05:30
stage: Analytics
2021-09-30 23:02:18 +05:30
group: Product Intelligence
2022-11-25 23:54:43 +05:30
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
2021-09-30 23:02:18 +05:30
---
# Service Ping Guide **(FREE SELF)**
2022-05-07 20:08:51 +05:30
> - Introduced in GitLab Ultimate 11.2, more statistics.
> - In GitLab 14.1, [renamed from Usage Ping to Service Ping](https://gitlab.com/groups/gitlab-org/-/epics/5990). In 14.0 and earlier, use the Usage Ping documentation for the Rails commands appropriate to your version.
2021-09-30 23:02:18 +05:30
2022-01-26 12:08:38 +05:30
Service Ping is a GitLab process that collects and sends a weekly payload to GitLab.
2021-09-30 23:02:18 +05:30
The payload provides important high-level data that helps our product, support,
2022-01-26 12:08:38 +05:30
and sales teams understand how GitLab is used. The data helps to:
2021-09-30 23:02:18 +05:30
- Compare counts month over month (or week over week) to get a rough sense for how an instance uses
different product features.
- Collect other facts that help us classify and understand GitLab installations.
2022-01-26 12:08:38 +05:30
- Calculate our stage monthly active users (SMAU), which helps to measure the success of our stages
2021-09-30 23:02:18 +05:30
and features.
2022-01-26 12:08:38 +05:30
Service Ping information is not anonymous. It's linked to the instance's hostname, but does
2021-09-30 23:02:18 +05:30
not contain project names, usernames, or any other specific data.
2022-08-27 11:52:29 +05:30
Service Ping is enabled by default. However, you can [disable](../../user/admin_area/settings/usage_statistics.md#enable-or-disable-usage-statistics) it on any self-managed instance. When Service Ping is enabled, GitLab gathers data from the other instances and can show your instance's usage statistics to your users.
2021-09-30 23:02:18 +05:30
2022-01-26 12:08:38 +05:30
## Service Ping terminology
2021-09-30 23:02:18 +05:30
We use the following terminology to describe the Service Ping components:
- **Service Ping**: the process that collects and generates a JSON payload.
- **Service Data**: the contents of the Service Ping JSON payload. This includes metrics.
- **Metrics**: primarily made up of row counts for different tables in an instance's database. Each
metric has a corresponding [metric definition](metrics_dictionary.md#metrics-definition-and-validation)
in a YAML file.
2021-12-11 22:18:48 +05:30
- **MAU**: monthly active users.
- **WAU**: weekly active users.
2021-09-30 23:02:18 +05:30
2021-12-11 22:18:48 +05:30
### Limitations
- Service Ping does not track frontend events things like page views, link clicks, or user sessions.
- Service Ping focuses only on aggregated backend events.
Because of these limitations we recommend you:
- Instrument your products with Snowplow for more detailed analytics on GitLab.com.
- Use Service Ping to track aggregated backend events on self-managed instances.
2021-09-30 23:02:18 +05:30
## Service Ping request flow
The following example shows a basic request/response flow between a GitLab instance, the Versions Application, the License Application, Salesforce, the GitLab S3 Bucket, the GitLab Snowflake Data Warehouse, and Sisense:
```mermaid
sequenceDiagram
participant GitLab Instance
participant Versions Application
participant Licenses Application
participant Salesforce
participant S3 Bucket
participant Snowflake DW
participant Sisense Dashboards
GitLab Instance->>Versions Application: Send Service Ping
loop Process usage data
Versions Application->>Versions Application: Parse usage data
Versions Application->>Versions Application: Write to database
Versions Application->>Versions Application: Update license ping time
end
loop Process data for Salesforce
Versions Application-xLicenses Application: Request Zuora subscription id
Licenses Application-xVersions Application: Zuora subscription id
Versions Application-xSalesforce: Request Zuora account id by Zuora subscription id
Salesforce-xVersions Application: Zuora account id
Versions Application-xSalesforce: Usage data for the Zuora account
end
Versions Application->>S3 Bucket: Export Versions database
S3 Bucket->>Snowflake DW: Import data
Snowflake DW->>Snowflake DW: Transform data using dbt
Snowflake DW->>Sisense Dashboards: Data available for querying
2021-12-11 22:18:48 +05:30
Versions Application->>GitLab Instance: DevOps Score (Conversational Development Index)
2021-09-30 23:02:18 +05:30
```
## How Service Ping works
2021-10-27 15:23:28 +05:30
1. The Service Ping [cron job](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/gitlab_service_ping_worker.rb#L24) is set in Sidekiq to run weekly.
2022-04-04 11:22:00 +05:30
1. When the cron job runs, it calls [`Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values)`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/service_ping/submit_service.rb).
1. `Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values)` [cascades down](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb) to ~400+ other counter method calls.
1. The response of all methods calls are [merged together](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb#L68) into a single JSON payload.
2021-10-27 15:23:28 +05:30
1. The JSON payload is then [posted to the Versions application](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/service_ping/submit_service.rb#L20)
2021-09-30 23:02:18 +05:30
If a firewall exception is needed, the required URL depends on several things. If
the hostname is `version.gitlab.com`, the protocol is `TCP`, and the port number is `443`,
the required URL is <https://version.gitlab.com/>.
2022-04-04 11:22:00 +05:30
1. In case of an error, it will be reported to the Version application along with following pieces of information:
2022-07-16 23:28:13 +05:30
- `uuid` - GitLab instance unique identifier
- `hostname` - GitLab instance hostname
- `version` - GitLab instance current versions
- `elapsed` - Amount of time which passed since Service Ping report process started and moment of error occurrence
- `message` - Error message
<pre>
<code>
{
"uuid"=>"02333324-1cd7-4c3b-a45b-a4993f05fb1d",
"hostname"=>"127.0.0.1",
"version"=>"14.7.0-pre",
"elapsed"=>0.006946,
"message"=>'PG::UndefinedColumn: ERROR: column \"non_existent_attribute\" does not exist\nLINE 1: SELECT COUNT(non_existent_attribute) FROM \"issues\" /*applica...'
}
</code>
</pre>
1. Finally, the timing metadata information that is used for diagnostic purposes is submitted to the Versions application. It consists of a list of metric identifiers and the time it took to calculate the metrics:
2022-08-27 11:52:29 +05:30
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37911) in GitLab 15.0 [with a flag](../../user/feature_flags.md), enabled by default.
2022-08-13 15:12:31 +05:30
> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/295289) in GitLab 15.2. [Feature flag `measure_service_ping_metric_collection`](https://gitlab.com/gitlab-org/gitlab/-/issues/358128) removed.
2022-07-16 23:28:13 +05:30
```ruby
2022-08-13 15:12:31 +05:30
{
"metadata"=>
{
"uuid"=>"0000000-0000-0000-0000-000000000000",
"metrics"=>
2022-07-16 23:28:13 +05:30
[{"name"=>"version", "time_elapsed"=>1.1811964213848114e-05},
{"name"=>"installation_type", "time_elapsed"=>0.00017242692410945892},
{"name"=>"license_billable_users", "time_elapsed"=>0.009520471096038818},
....
{"name"=>"counts.clusters_platforms_eks",
"time_elapsed"=>0.05638605775311589},
{"name"=>"counts.clusters_platforms_gke",
"time_elapsed"=>0.40995341585949063},
{"name"=>"counts.clusters_platforms_user",
"time_elapsed"=>0.06410990096628666},
{"name"=>"counts.clusters_management_project",
2022-08-13 15:12:31 +05:30
"time_elapsed"=>0.24020783510059118}
2022-07-16 23:28:13 +05:30
]
}
}
```
2021-09-30 23:02:18 +05:30
2021-10-27 15:23:28 +05:30
### On a Geo secondary site
We also collect metrics specific to [Geo](../../administration/geo/index.md) secondary sites to send with Service Ping.
1. The [Geo secondary service ping cron job](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/geo/secondary_usage_data_cron_worker.rb) is set in Sidekiq to run weekly.
1. When the cron job runs, it calls [`SecondaryUsageData.update_metrics!`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/geo/secondary_usage_data.rb#L33). This collects the relevant metrics from Prometheus and stores the data in the Geo secondary tracking database for transmission to the primary site during a [Geo node status update](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/geo_node_status.rb#L105).
1. Geo node status data is sent with the JSON payload in the process described above. The following is an example of the payload where each object in the array represents a Geo node:
```json
[
{
"repository_verification_enabled"=>true,
"repositories_replication_enabled"=>true,
"repositories_synced_count"=>24,
"repositories_failed_count"=>0,
"git_fetch_event_count_weekly"=>nil,
"git_push_event_count_weekly"=>nil,
... other geo node status fields
}
]
2021-09-30 23:02:18 +05:30
```
## Implementing Service Ping
2021-11-11 11:23:49 +05:30
See the [implement Service Ping](implement.md) guide.
2021-09-30 23:02:18 +05:30
## Example Service Ping payload
The following is example content of the Service Ping payload.
```json
{
"uuid": "0000000-0000-0000-0000-000000000000",
"hostname": "example.com",
"version": "12.10.0-pre",
"installation_type": "omnibus-gitlab",
"active_user_count": 999,
"recorded_at": "2020-04-17T07:43:54.162+00:00",
"edition": "EEU",
"license_md5": "00000000000000000000000000000000",
2022-08-13 15:12:31 +05:30
"license_sha256: "0000000000000000000000000000000000000000000000000000000000000000",
2021-09-30 23:02:18 +05:30
"license_id": null,
"historical_max_users": 999,
"licensee": {
"Name": "ABC, Inc.",
"Email": "email@example.com",
"Company": "ABC, Inc."
},
"license_user_count": 999,
"license_starts_at": "2020-01-01",
"license_expires_at": "2021-01-01",
"license_plan": "ultimate",
"license_add_ons": {
},
"license_trial": false,
"counts": {
"assignee_lists": 999,
"boards": 999,
"ci_builds": 999,
...
},
"container_registry_enabled": true,
"dependency_proxy_enabled": false,
"gitlab_shared_runners_enabled": true,
"gravatar_enabled": true,
"influxdb_metrics_enabled": true,
"ldap_enabled": false,
"mattermost_enabled": false,
"omniauth_enabled": true,
"prometheus_enabled": false,
"prometheus_metrics_enabled": false,
"reply_by_email_enabled": "incoming+%{key}@incoming.gitlab.com",
"signup_enabled": true,
"web_ide_clientside_preview_enabled": true,
"projects_with_expiration_policy_disabled": 999,
"projects_with_expiration_policy_enabled": 999,
...
"elasticsearch_enabled": true,
"license_trial_ends_on": null,
"geo_enabled": false,
"git": {
"version": {
"major": 2,
"minor": 26,
"patch": 1
}
},
"gitaly": {
"version": "12.10.0-rc1-93-g40980d40",
"servers": 56,
"clusters": 14,
"filesystems": [
"EXT_2_3_4"
]
},
"gitlab_pages": {
"enabled": true,
"version": "1.17.0"
},
"container_registry_server": {
"vendor": "gitlab",
"version": "2.9.1-gitlab"
},
"database": {
"adapter": "postgresql",
"version": "9.6.15",
2022-03-02 08:16:31 +05:30
"pg_system_id": 6842684531675334351,
"flavor": "Cloud SQL for PostgreSQL"
2021-09-30 23:02:18 +05:30
},
"analytics_unique_visits": {
"g_analytics_contribution": 999,
...
},
"usage_activity_by_stage": {
"configure": {
"project_clusters_enabled": 999,
...
},
"create": {
"merge_requests": 999,
...
},
"manage": {
"events": 999,
...
},
"monitor": {
"clusters": 999,
...
},
"package": {
"projects_with_packages": 999
},
"plan": {
"issues": 999,
...
},
"release": {
"deployments": 999,
...
},
"secure": {
"user_container_scanning_jobs": 999,
...
},
"verify": {
"ci_builds": 999,
...
}
},
"usage_activity_by_stage_monthly": {
"configure": {
"project_clusters_enabled": 999,
...
},
"create": {
"merge_requests": 999,
...
},
"manage": {
"events": 999,
...
},
"monitor": {
"clusters": 999,
...
},
"package": {
"projects_with_packages": 999
},
"plan": {
"issues": 999,
...
},
"release": {
"deployments": 999,
...
},
"secure": {
"user_container_scanning_jobs": 999,
...
},
"verify": {
"ci_builds": 999,
...
}
},
"topology": {
"duration_s": 0.013836685999194742,
"application_requests_per_hour": 4224,
"query_apdex_weekly_average": 0.996,
"failures": [],
"nodes": [
{
"node_memory_total_bytes": 33269903360,
"node_memory_utilization": 0.35,
"node_cpus": 16,
"node_cpu_utilization": 0.2,
"node_uname_info": {
"machine": "x86_64",
"sysname": "Linux",
"release": "4.19.76-linuxkit"
},
"node_services": [
{
"name": "web",
"process_count": 16,
"process_memory_pss": 233349888,
"process_memory_rss": 788220927,
"process_memory_uss": 195295487,
"server": "puma"
},
{
"name": "sidekiq",
"process_count": 1,
"process_memory_pss": 734080000,
"process_memory_rss": 750051328,
"process_memory_uss": 731533312
},
...
],
...
},
...
]
}
}
```
## Notable changes
2022-03-02 08:16:31 +05:30
In GitLab 14.6, [`flavor`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75587) was added to try to detect the underlying managed database variant.
Possible values are "Amazon Aurora PostgreSQL", "PostgreSQL on Amazon RDS", "Cloud SQL for PostgreSQL",
"Azure Database for PostgreSQL - Flexible Server", or "null".
2021-09-30 23:02:18 +05:30
In GitLab 13.5, `pg_system_id` was added to send the [PostgreSQL system identifier](https://www.2ndquadrant.com/en/blog/support-for-postgresqls-system-identifier-in-barman/).
2022-10-11 01:57:18 +05:30
## Export Service Ping data
2021-09-30 23:02:18 +05:30
2022-10-11 01:57:18 +05:30
Rake tasks exist to export Service Ping data in different formats.
2021-09-30 23:02:18 +05:30
- The Rake tasks export the raw SQL queries for `count`, `distinct_count`, `sum`.
- The Rake tasks export the Redis counter class or the line of the Redis block for `redis_usage_data`.
- The Rake tasks calculate the `alt_usage_data` metrics.
In the home directory of your local GitLab installation run the following Rake tasks for the YAML and JSON versions respectively:
```shell
2022-10-11 01:57:18 +05:30
# for YAML export of SQL queries
2021-09-30 23:02:18 +05:30
bin/rake gitlab:usage_data:dump_sql_in_yaml
2022-10-11 01:57:18 +05:30
# for JSON export of SQL queries
2021-09-30 23:02:18 +05:30
bin/rake gitlab:usage_data:dump_sql_in_json
2022-10-11 01:57:18 +05:30
# for JSON export of Non SQL data
bin/rake gitlab:usage_data:dump_non_sql_in_json
2021-09-30 23:02:18 +05:30
# You may pipe the output into a file
bin/rake gitlab:usage_data:dump_sql_in_yaml > ~/Desktop/usage-metrics-2020-09-02.yaml
```
2021-12-11 22:18:48 +05:30
## Generate Service Ping
2021-09-30 23:02:18 +05:30
2021-12-11 22:18:48 +05:30
To generate Service Ping, use [Teleport](https://goteleport.com/docs/) or a detached screen session on a remote server.
2021-09-30 23:02:18 +05:30
### Triggering
2021-12-11 22:18:48 +05:30
#### Trigger Service Ping with Teleport
1. Request temporary [access](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Rails_Console_via_Teleport.md#how-to-use-teleport-to-connect-to-rails-console) to the required environment.
1. After your approval is issued, [access the Rails console](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Rails_Console_via_Teleport.md#access-approval).
2022-10-11 01:57:18 +05:30
1. Run `GitlabServicePingWorker.new.perform('triggered_from_cron' => false)`.
2021-12-11 22:18:48 +05:30
#### Trigger Service Ping with a detached screen session
1. Connect to bastion with agent forwarding:
2022-01-26 12:08:38 +05:30
```shell
ssh -A lb-bastion.gprd.gitlab.com
```
2021-12-11 22:18:48 +05:30
1. Create named screen:
2022-01-26 12:08:38 +05:30
```shell
screen -S <username>_usage_ping_<date>
```
2021-12-11 22:18:48 +05:30
1. Connect to console host:
2022-07-16 23:28:13 +05:30
2022-01-26 12:08:38 +05:30
```shell
ssh $USER-rails@console-01-sv-gprd.c.gitlab-production.internal
```
2021-12-11 22:18:48 +05:30
2022-01-26 12:08:38 +05:30
1. Run:
2021-12-11 22:18:48 +05:30
2022-01-26 12:08:38 +05:30
```shell
2022-10-11 01:57:18 +05:30
GitlabServicePingWorker.new.perform('triggered_from_cron' => false)
2022-01-26 12:08:38 +05:30
```
2021-12-11 22:18:48 +05:30
2022-01-26 12:08:38 +05:30
1. To detach from screen, press `ctrl + A`, `ctrl + D`.
1. Exit from bastion:
2022-07-16 23:28:13 +05:30
2022-01-26 12:08:38 +05:30
```shell
exit
```
2021-09-30 23:02:18 +05:30
2022-07-16 23:28:13 +05:30
1. Get the metrics duration from logs:
2022-07-23 23:45:48 +05:30
Search in Google Console logs for `time_elapsed`. [Query example](https://cloudlogging.app.goo.gl/nWheZvD8D3nWazNe6).
2022-07-16 23:28:13 +05:30
2021-09-30 23:02:18 +05:30
### Verification (After approx 30 hours)
2022-04-04 11:22:00 +05:30
#### Verify with Teleport
2021-12-11 22:18:48 +05:30
1. Follow [the steps](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Rails_Console_via_Teleport.md#how-to-use-teleport-to-connect-to-rails-console) to request a new access to the required environment and connect to the Rails console
1. Check the last payload in `raw_usage_data` table: `RawUsageData.last.payload`
1. Check the when the payload was sent: `RawUsageData.last.sent_at`
#### Verify using detached screen session
1. Reconnect to bastion:
2022-01-26 12:08:38 +05:30
```shell
ssh -A lb-bastion.gprd.gitlab.com
```
2021-12-11 22:18:48 +05:30
1. Find your screen session:
2022-01-26 12:08:38 +05:30
```shell
screen -ls
```
2021-12-11 22:18:48 +05:30
1. Attach to your screen session:
2022-01-26 12:08:38 +05:30
```shell
screen -x 14226.mwawrzyniak_usage_ping_2021_01_22
```
1. Check the last payload in `raw_usage_data` table:
2022-07-16 23:28:13 +05:30
2022-01-26 12:08:38 +05:30
```shell
RawUsageData.last.payload
```
1. Check the when the payload was sent:
```shell
RawUsageData.last.sent_at
```
### Skip database write operations
To skip database write operations, DevOps report creation, and storage of usage data payload, pass an optional argument:
```shell
skip_db_write:
2022-10-11 01:57:18 +05:30
GitlabServicePingWorker.new.perform('triggered_from_cron' => false, 'skip_db_write' => true)
2022-01-26 12:08:38 +05:30
```
2021-09-30 23:02:18 +05:30
2022-04-04 11:22:00 +05:30
## Monitoring
2022-05-07 20:08:51 +05:30
Service Ping reporting process state is monitored with [internal SiSense dashboard](https://app.periscopedata.com/app/gitlab/968489/Product-Intelligence---Service-Ping-Health).
2022-04-04 11:22:00 +05:30
2022-01-26 12:08:38 +05:30
## Related topics
- [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/)
- [Snowplow Guide](../snowplow/index.md)
- [Product Intelligence Direction](https://about.gitlab.com/direction/product-intelligence/)
- [Data Analysis Process](https://about.gitlab.com/handbook/business-technology/data-team/#data-analysis-process/)
- [Data for Product Managers](https://about.gitlab.com/handbook/business-technology/data-team/programs/data-for-product-managers/)
- [Data Infrastructure](https://about.gitlab.com/handbook/business-technology/data-team/platform/infrastructure/)