debian-mirror-gitlab/doc/ci/testing/browser_performance_testing.md

---
stage: Verify
group: Pipeline Execution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---

<!--- start_remove The following content will be removed on remove_date: '2024-05-22' -->
# Browser Performance Testing **(PREMIUM)**

> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3507) in GitLab 10.3.

WARNING:
This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/388719) in GitLab 15.9
and is planned for removal in 17.0. This change is a breaking change.

If your application offers a web interface and you're using
[GitLab CI/CD](../index.md), you can quickly determine the rendering performance
impact of pending code changes in the browser.

NOTE:
You can automate this feature in your applications by using [Auto DevOps](../../topics/autodevops/index.md).

## Overview

GitLab uses [Sitespeed.io](https://www.sitespeed.io), a free and open source
tool, for measuring the rendering performance of web sites. The
[Sitespeed plugin](https://gitlab.com/gitlab-org/gl-performance) that GitLab built outputs
the performance score for each page analyzed in a file called `browser-performance.json`
this data can be shown on Merge Requests.

## Use cases

Consider the following workflow:

1. A member of the marketing team is attempting to track engagement by adding a new tool.
1. With browser performance metrics, they see how their changes are impacting the usability
   of the page for end users.
1. The metrics show that after their changes, the performance score of the page has gone down.
1. When looking at the detailed report, they see the new JavaScript library was
   included in `<head>`, which affects loading page speed.
1. They ask for help from a front end developer, who sets the library to load asynchronously.
1. The frontend developer approves the merge request, and authorizes its deployment to production.

## How browser performance testing works

First, define a job in your `.gitlab-ci.yml` file that generates the
[Browser Performance report artifact](../yaml/artifacts_reports.md#artifactsreportsbrowser_performance).
GitLab then checks this report, compares key performance metrics for each page
between the source and target branches, and shows the information in the merge request.

For an example Browser Performance job, see
[Configuring Browser Performance Testing](#configuring-browser-performance-testing).

NOTE:
If the Browser Performance report has no data to compare, such as when you add the
Browser Performance job in your `.gitlab-ci.yml` for the very first time,
the Browser Performance report widget doesn't display. It must have run at least
once on the target branch (`main`, for example), before it displays in a
merge request targeting that branch.

![Browser Performance Widget](img/browser_performance_testing.png)

## Configuring Browser Performance Testing

This example shows how to run the [sitespeed.io container](https://hub.docker.com/r/sitespeedio/sitespeed.io/)
on your code by using GitLab CI/CD and [sitespeed.io](https://www.sitespeed.io)
using Docker-in-Docker.

1. First, set up GitLab Runner with a
   [Docker-in-Docker build](../docker/using_docker_build.md#use-docker-in-docker).
1. Configure the default Browser Performance Testing CI/CD job as follows in your `.gitlab-ci.yml` file:

   ```yaml
   include:
     template: Verify/Browser-Performance.gitlab-ci.yml

   browser_performance:
     variables:
       URL: https://example.com
   ```

WARNING:
In GitLab 13.12 and earlier, the job [was named](https://gitlab.com/gitlab-org/gitlab/-/issues/225914) `performance`.

The above example:

- Creates a `browser_performance` job in your CI/CD pipeline and runs sitespeed.io against the webpage you
  defined in `URL` to gather key metrics.
- Uses a template that doesn't work with Kubernetes clusters. If you are using a Kubernetes cluster,
  use [`template: Jobs/Browser-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Browser-Performance-Testing.gitlab-ci.yml)
   instead.
- Uses a CI/CD template that is included in all GitLab installations since 12.4. If you are using
  GitLab 12.3 or earlier, you must [add the configuration manually](#gitlab-versions-132-and-earlier).

The template uses the [GitLab plugin for sitespeed.io](https://gitlab.com/gitlab-org/gl-performance),
and it saves the full HTML sitespeed.io report as a [Browser Performance report artifact](../yaml/artifacts_reports.md#artifactsreportsbrowser_performance)
that you can later download and analyze. This implementation always takes the latest
Browser Performance artifact available. If [GitLab Pages](../../user/project/pages/index.md) is enabled,
you can view the report directly in your browser.

You can also customize the jobs with CI/CD variables:

- `SITESPEED_IMAGE`: Configure the Docker image to use for the job (default `sitespeedio/sitespeed.io`), but not the image version.
- `SITESPEED_VERSION`: Configure the version of the Docker image to use for the job (default `14.1.0`).
- `SITESPEED_OPTIONS`: Configure any additional sitespeed.io options as required (default `nil`). Refer to the [sitespeed.io documentation](https://www.sitespeed.io/documentation/sitespeed.io/configuration/) for more details.

For example, you can override the number of runs sitespeed.io
makes on the given URL, and change the version:

```yaml
include:
  template: Verify/Browser-Performance.gitlab-ci.yml

browser_performance:
  variables:
    URL: https://www.sitespeed.io/
    SITESPEED_VERSION: 13.2.0
    SITESPEED_OPTIONS: -n 5
```

### Configuring degradation threshold

> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27599) in GitLab 13.0.

You can configure the sensitivity of degradation alerts to avoid getting alerts for minor drops in metrics.
This is done by setting the `DEGRADATION_THRESHOLD` CI/CD variable. In the example below, the alert only shows up
if the `Total Score` metric degrades by 5 points or more:

```yaml
include:
  template: Verify/Browser-Performance.gitlab-ci.yml

browser_performance:
  variables:
    URL: https://example.com
    DEGRADATION_THRESHOLD: 5
```

The `Total Score` metric is based on sitespeed.io's [coach performance score](https://www.sitespeed.io/documentation/sitespeed.io/metrics/#performance-score). There is more information in [the coach documentation](https://www.sitespeed.io/documentation/coach/how-to/#what-do-the-coach-do).

### Performance testing on Review Apps

The above CI YAML configuration is great for testing against static environments, and it can
be extended for dynamic environments, but a few extra steps are required:

1. The `browser_performance` job should run after the dynamic environment has started.
1. In the `review` job:
    1. Generate a URL list file with the dynamic URL.
    1. Save the file as an artifact, for example with `echo $CI_ENVIRONMENT_URL > environment_url.txt`
    in your job's `script`.
    1. Pass the list as the URL environment variable (which can be a URL or a file containing URLs)
    to the `browser_performance` job.
1. You can now run the sitespeed.io container against the desired hostname and
   paths.

Your `.gitlab-ci.yml` file would look like:

```yaml
stages:
  - deploy
  - performance

include:
  template: Verify/Browser-Performance.gitlab-ci.yml

review:
  stage: deploy
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    url: http://$CI_COMMIT_REF_SLUG.$APPS_DOMAIN
  script:
    - run_deploy_script
    - echo $CI_ENVIRONMENT_URL > environment_url.txt
  artifacts:
    paths:
      - environment_url.txt
  only:
    - branches
  except:
    - master

browser_performance:
  dependencies:
    - review
  variables:
    URL: environment_url.txt
```

### GitLab versions 13.2 and earlier

Browser Performance Testing has gone through several changes since its introduction.
In this section we detail these changes and how you can run the test based on your
GitLab version:

- In 13.2 the feature was renamed from `Performance` to `Browser Performance` with additional
  template CI/CD variables.
- In GitLab 12.4 [a job template was made available](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Verify/Browser-Performance.gitlab-ci.yml).
- For 11.5 to 12.3 no template is available and the job has to be defined manually as follows:

  ```yaml
  performance:
    stage: performance
    image: docker:git
    variables:
      URL: https://example.com
      SITESPEED_VERSION: 14.1.0
      SITESPEED_OPTIONS: ''
    services:
      - docker:stable-dind
    script:
      - mkdir gitlab-exporter
      - wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/1.1.0/index.js
      - mkdir sitespeed-results
      - docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:$SITESPEED_VERSION --plugins.add ./gitlab-exporter --outputFolder sitespeed-results $URL $SITESPEED_OPTIONS
      - mv sitespeed-results/data/performance.json performance.json
    artifacts:
      paths:
        - performance.json
        - sitespeed-results/
      reports:
        performance: performance.json
  ```

- For 11.4 and earlier the job should be defined as follows:

  ```yaml
  performance:
    stage: performance
    image: docker:git
    variables:
      URL: https://example.com
    services:
      - docker:stable-dind
    script:
      - mkdir gitlab-exporter
      - wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/1.1.0/index.js
      - mkdir sitespeed-results
      - docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:6.3.1 --plugins.add ./gitlab-exporter --outputFolder sitespeed-results $URL
      - mv sitespeed-results/data/performance.json performance.json
    artifacts:
      paths:
        - performance.json
        - sitespeed-results/
  ```

Upgrading to the latest version and using the templates is recommended, to ensure
you receive the latest updates, including updates to the sitespeed.io versions.
New upstream version 15.2.2+ds1 2022-08-13 15:12:31 +05:30			`---`
			`stage: Verify`
New upstream version 15.10.7+ds1 2023-05-27 22:25:52 +05:30			`group: Pipeline Execution`
New upstream version 15.5.4+ds1 2022-11-25 23:54:43 +05:30			`info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments`
New upstream version 15.2.2+ds1 2022-08-13 15:12:31 +05:30			`---`

New upstream version 15.11.6+ds1 2023-06-20 00:43:36 +05:30			`<!--- start_remove The following content will be removed on remove_date: '2024-05-22' -->`
New upstream version 15.2.2+ds1 2022-08-13 15:12:31 +05:30			`# Browser Performance Testing (PREMIUM)`

			`> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3507) in GitLab 10.3.`

New upstream version 15.11.6+ds1 2023-06-20 00:43:36 +05:30			`WARNING:`
			`This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/388719) in GitLab 15.9`
			`and is planned for removal in 17.0. This change is a breaking change.`

New upstream version 15.2.2+ds1 2022-08-13 15:12:31 +05:30			`If your application offers a web interface and you're using`
			`[GitLab CI/CD](../index.md), you can quickly determine the rendering performance`
			`impact of pending code changes in the browser.`

			`NOTE:`
			`You can automate this feature in your applications by using [Auto DevOps](../../topics/autodevops/index.md).`

			`## Overview`

			`GitLab uses [Sitespeed.io](https://www.sitespeed.io), a free and open source`
			`tool, for measuring the rendering performance of web sites. The`
			`[Sitespeed plugin](https://gitlab.com/gitlab-org/gl-performance) that GitLab built outputs`
			the performance score for each page analyzed in a file called `browser-performance.json`
			`this data can be shown on Merge Requests.`

			`## Use cases`

			`Consider the following workflow:`

			`1. A member of the marketing team is attempting to track engagement by adding a new tool.`
			`1. With browser performance metrics, they see how their changes are impacting the usability`
			`of the page for end users.`
			`1. The metrics show that after their changes, the performance score of the page has gone down.`
			`1. When looking at the detailed report, they see the new JavaScript library was`
			included in `<head>`, which affects loading page speed.
			`1. They ask for help from a front end developer, who sets the library to load asynchronously.`
			`1. The frontend developer approves the merge request, and authorizes its deployment to production.`

			`## How browser performance testing works`

			First, define a job in your `.gitlab-ci.yml` file that generates the
			`[Browser Performance report artifact](../yaml/artifacts_reports.md#artifactsreportsbrowser_performance).`
			`GitLab then checks this report, compares key performance metrics for each page`
			`between the source and target branches, and shows the information in the merge request.`

			`For an example Browser Performance job, see`
			`[Configuring Browser Performance Testing](#configuring-browser-performance-testing).`

			`NOTE:`
			`If the Browser Performance report has no data to compare, such as when you add the`
			Browser Performance job in your `.gitlab-ci.yml` for the very first time,
			`the Browser Performance report widget doesn't display. It must have run at least`
			once on the target branch (`main`, for example), before it displays in a
			`merge request targeting that branch.`

			`![Browser Performance Widget](img/browser_performance_testing.png)`

			`## Configuring Browser Performance Testing`

			`This example shows how to run the [sitespeed.io container](https://hub.docker.com/r/sitespeedio/sitespeed.io/)`
			`on your code by using GitLab CI/CD and [sitespeed.io](https://www.sitespeed.io)`
			`using Docker-in-Docker.`

			`1. First, set up GitLab Runner with a`
			`[Docker-in-Docker build](../docker/using_docker_build.md#use-docker-in-docker).`
			1. Configure the default Browser Performance Testing CI/CD job as follows in your `.gitlab-ci.yml` file:

			```yaml
			`include:`
			`template: Verify/Browser-Performance.gitlab-ci.yml`

			`browser_performance:`
			`variables:`
			`URL: https://example.com`
			```

			`WARNING:`
			In GitLab 13.12 and earlier, the job [was named](https://gitlab.com/gitlab-org/gitlab/-/issues/225914) `performance`.

			`The above example:`

			- Creates a `browser_performance` job in your CI/CD pipeline and runs sitespeed.io against the webpage you
			defined in `URL` to gather key metrics.
			`- Uses a template that doesn't work with Kubernetes clusters. If you are using a Kubernetes cluster,`
			use [`template: Jobs/Browser-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Browser-Performance-Testing.gitlab-ci.yml)
			`instead.`
			`- Uses a CI/CD template that is included in all GitLab installations since 12.4. If you are using`
			`GitLab 12.3 or earlier, you must [add the configuration manually](#gitlab-versions-132-and-earlier).`

			`The template uses the [GitLab plugin for sitespeed.io](https://gitlab.com/gitlab-org/gl-performance),`
			`and it saves the full HTML sitespeed.io report as a [Browser Performance report artifact](../yaml/artifacts_reports.md#artifactsreportsbrowser_performance)`
			`that you can later download and analyze. This implementation always takes the latest`
			`Browser Performance artifact available. If [GitLab Pages](../../user/project/pages/index.md) is enabled,`
			`you can view the report directly in your browser.`

			`You can also customize the jobs with CI/CD variables:`

			- `SITESPEED_IMAGE`: Configure the Docker image to use for the job (default `sitespeedio/sitespeed.io`), but not the image version.
			- `SITESPEED_VERSION`: Configure the version of the Docker image to use for the job (default `14.1.0`).
			- `SITESPEED_OPTIONS`: Configure any additional sitespeed.io options as required (default `nil`). Refer to the [sitespeed.io documentation](https://www.sitespeed.io/documentation/sitespeed.io/configuration/) for more details.

			`For example, you can override the number of runs sitespeed.io`
			`makes on the given URL, and change the version:`

			```yaml
			`include:`
			`template: Verify/Browser-Performance.gitlab-ci.yml`

			`browser_performance:`
			`variables:`
			`URL: https://www.sitespeed.io/`
			`SITESPEED_VERSION: 13.2.0`
			`SITESPEED_OPTIONS: -n 5`
			```

			`### Configuring degradation threshold`

			`> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27599) in GitLab 13.0.`

			`You can configure the sensitivity of degradation alerts to avoid getting alerts for minor drops in metrics.`
			This is done by setting the `DEGRADATION_THRESHOLD` CI/CD variable. In the example below, the alert only shows up
			if the `Total Score` metric degrades by 5 points or more:

			```yaml
			`include:`
			`template: Verify/Browser-Performance.gitlab-ci.yml`

			`browser_performance:`
			`variables:`
			`URL: https://example.com`
			`DEGRADATION_THRESHOLD: 5`
			```

			The `Total Score` metric is based on sitespeed.io's [coach performance score](https://www.sitespeed.io/documentation/sitespeed.io/metrics/#performance-score). There is more information in [the coach documentation](https://www.sitespeed.io/documentation/coach/how-to/#what-do-the-coach-do).

			`### Performance testing on Review Apps`

			`The above CI YAML configuration is great for testing against static environments, and it can`
			`be extended for dynamic environments, but a few extra steps are required:`

			1. The `browser_performance` job should run after the dynamic environment has started.
			1. In the `review` job:
			`1. Generate a URL list file with the dynamic URL.`
			1. Save the file as an artifact, for example with `echo $CI_ENVIRONMENT_URL > environment_url.txt`
			in your job's `script`.
			`1. Pass the list as the URL environment variable (which can be a URL or a file containing URLs)`
			to the `browser_performance` job.
			`1. You can now run the sitespeed.io container against the desired hostname and`
			`paths.`

			Your `.gitlab-ci.yml` file would look like:

			```yaml
			`stages:`
			`- deploy`
			`- performance`

			`include:`
			`template: Verify/Browser-Performance.gitlab-ci.yml`

			`review:`
			`stage: deploy`
			`environment:`
			`name: review/$CI_COMMIT_REF_SLUG`
			`url: http://$CI_COMMIT_REF_SLUG.$APPS_DOMAIN`
			`script:`
			`- run_deploy_script`
			`- echo $CI_ENVIRONMENT_URL > environment_url.txt`
			`artifacts:`
			`paths:`
			`- environment_url.txt`
			`only:`
			`- branches`
			`except:`
			`- master`

			`browser_performance:`
			`dependencies:`
			`- review`
			`variables:`
			`URL: environment_url.txt`
			```

			`### GitLab versions 13.2 and earlier`

			`Browser Performance Testing has gone through several changes since its introduction.`
			`In this section we detail these changes and how you can run the test based on your`
			`GitLab version:`

			- In 13.2 the feature was renamed from `Performance` to `Browser Performance` with additional
			`template CI/CD variables.`
			`- In GitLab 12.4 [a job template was made available](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Verify/Browser-Performance.gitlab-ci.yml).`
			`- For 11.5 to 12.3 no template is available and the job has to be defined manually as follows:`

			```yaml
			`performance:`
			`stage: performance`
			`image: docker:git`
			`variables:`
			`URL: https://example.com`
			`SITESPEED_VERSION: 14.1.0`
			`SITESPEED_OPTIONS: ''`
			`services:`
			`- docker:stable-dind`
			`script:`
			`- mkdir gitlab-exporter`
			`- wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/1.1.0/index.js`
			`- mkdir sitespeed-results`
			`- docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:$SITESPEED_VERSION --plugins.add ./gitlab-exporter --outputFolder sitespeed-results $URL $SITESPEED_OPTIONS`
			`- mv sitespeed-results/data/performance.json performance.json`
			`artifacts:`
			`paths:`
			`- performance.json`
			`- sitespeed-results/`
			`reports:`
			`performance: performance.json`
			```

			`- For 11.4 and earlier the job should be defined as follows:`

			```yaml
			`performance:`
			`stage: performance`
			`image: docker:git`
			`variables:`
			`URL: https://example.com`
			`services:`
			`- docker:stable-dind`
			`script:`
			`- mkdir gitlab-exporter`
			`- wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/1.1.0/index.js`
			`- mkdir sitespeed-results`
			`- docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:6.3.1 --plugins.add ./gitlab-exporter --outputFolder sitespeed-results $URL`
			`- mv sitespeed-results/data/performance.json performance.json`
			`artifacts:`
			`paths:`
			`- performance.json`
			`- sitespeed-results/`
			```

			`Upgrading to the latest version and using the templates is recommended, to ensure`
			`you receive the latest updates, including updates to the sitespeed.io versions.`