202 lines
9.2 KiB
Markdown
202 lines
9.2 KiB
Markdown
---
|
|
stage: Verify
|
|
group: Testing
|
|
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
|
|
type: reference, howto
|
|
---
|
|
|
|
# Load Performance Testing **(PREMIUM)**
|
|
|
|
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10683) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2.
|
|
|
|
With Load Performance Testing, you can test the impact of any pending code changes
|
|
to your application's backend in [GitLab CI/CD](../../../ci/README.md).
|
|
|
|
GitLab uses [k6](https://k6.io/), a free and open source
|
|
tool, for measuring the system performance of applications under
|
|
load.
|
|
|
|
Unlike [Browser Performance Testing](browser_performance_testing.md), which is
|
|
used to measure how web sites perform in client browsers, Load Performance Testing
|
|
can be used to perform various types of [load tests](https://k6.io/docs/#use-cases)
|
|
against application endpoints such as APIs, Web Controllers, and so on.
|
|
This can be used to test how the backend or the server performs at scale.
|
|
|
|
For example, you can use Load Performance Testing to perform many concurrent
|
|
GET calls to a popular API endpoint in your application to see how it performs.
|
|
|
|
## How Load Performance Testing works
|
|
|
|
First, define a job in your `.gitlab-ci.yml` file that generates the
|
|
[Load Performance report artifact](../../../ci/pipelines/job_artifacts.md#artifactsreportsload_performance).
|
|
GitLab checks this report, compares key load performance metrics
|
|
between the source and target branches, and then shows the information in a merge request widget:
|
|
|
|
![Load Performance Widget](img/load_performance_testing.png)
|
|
|
|
Next, you need to configure the test environment and write the k6 test.
|
|
|
|
The key performance metrics that the merge request widget shows after the test completes are:
|
|
|
|
- Checks: The percentage pass rate of the [checks](https://k6.io/docs/using-k6/checks) configured in the k6 test.
|
|
- TTFB P90: The 90th percentile of how long it took to start receiving responses, aka the [Time to First Byte](https://en.wikipedia.org/wiki/Time_to_first_byte) (TTFB).
|
|
- TTFB P95: The 95th percentile for TTFB.
|
|
- RPS: The average requests per second (RPS) rate the test was able to achieve.
|
|
|
|
NOTE: **Note:**
|
|
If the Load Performance report has no data to compare, such as when you add the
|
|
Load Performance job in your `.gitlab-ci.yml` for the very first time,
|
|
the Load Performance report widget won't show. It must have run at least
|
|
once on the target branch (`master`, for example), before it will display in a
|
|
merge request targeting that branch.
|
|
|
|
## Configure the Load Performance Testing job
|
|
|
|
Configuring your Load Performance Testing job can be broken down into several distinct parts:
|
|
|
|
- Determine the test parameters such as throughput, and so on.
|
|
- Set up the target test environment for load performance testing.
|
|
- Design and write the k6 test.
|
|
|
|
### Determine the test parameters
|
|
|
|
The first thing you need to do is determine the [type of load test](https://k6.io/docs/test-types/introduction)
|
|
you want to run, and how it will run (for example, the number of users, throughput, and so on).
|
|
|
|
Refer to the [k6 docs](https://k6.io/docs/), especially the [k6 testing guides](https://k6.io/docs/testing-guides),
|
|
for guidance on the above and more.
|
|
|
|
### Test Environment setup
|
|
|
|
A large part of the effort around load performance testing is to prepare the target test environment
|
|
for high loads. You should ensure it's able to handle the
|
|
[throughput](https://k6.io/blog/monthly-visits-concurrent-users) it will be tested with.
|
|
|
|
It's also typically required to have representative test data in the target environment
|
|
for the load performance test to use.
|
|
|
|
We strongly recommend [not running these tests against a production environment](https://k6.io/our-beliefs#load-test-in-a-pre-production-environment).
|
|
|
|
### Write the load performance test
|
|
|
|
After the environment is prepared, you can write the k6 test itself. k6 is a flexible
|
|
tool and can be used to run [many kinds of performance tests](https://k6.io/docs/test-types/introduction).
|
|
Refer to the [k6 documentation](https://k6.io/docs/) for detailed information on how to write tests.
|
|
|
|
### Configure the test in GitLab CI/CD
|
|
|
|
When your k6 test is ready, the next step is to configure the load performance
|
|
testing job in GitLab CI/CD. The easiest way to do this is to use the
|
|
[`Verify/Load-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Verify/Load-Performance-Testing.gitlab-ci.yml)
|
|
template that is included with GitLab.
|
|
|
|
NOTE: **Note:**
|
|
For large scale k6 tests you need to ensure the GitLab Runner instance performing the actual
|
|
test is able to handle running the test. Refer to [k6's guidance](https://k6.io/docs/testing-guides/running-large-tests#hardware-considerations)
|
|
for spec details. The [default shared GitLab.com runners](../../gitlab_com/#linux-shared-runners)
|
|
likely have insufficient specs to handle most large k6 tests.
|
|
|
|
This template runs the
|
|
[k6 Docker container](https://hub.docker.com/r/loadimpact/k6/) in the job and provides several ways to customize the
|
|
job.
|
|
|
|
An example configuration workflow:
|
|
|
|
1. Set up GitLab Runner to run Docker containers, like the
|
|
[Docker-in-Docker workflow](../../../ci/docker/using_docker_build.md#use-docker-in-docker-workflow-with-docker-executor).
|
|
1. Configure the default Load Performance Testing CI job in your `.gitlab-ci.yml` file.
|
|
You need to include the template and configure it with variables:
|
|
|
|
```yaml
|
|
include:
|
|
template: Verify/Load-Performance-Testing.gitlab-ci.yml
|
|
|
|
load_performance:
|
|
variables:
|
|
K6_TEST_FILE: <PATH TO K6 TEST FILE IN PROJECT>
|
|
```
|
|
|
|
The above example creates a `load_performance` job in your CI/CD pipeline that runs
|
|
the k6 test.
|
|
|
|
NOTE: **Note:**
|
|
For Kubernetes setups a different template should be used: [`Jobs/Load-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Load-Performance-Testing.gitlab-ci.yml).
|
|
|
|
k6 has [various options](https://k6.io/docs/using-k6/options) to configure how it will run tests, such as what throughput (RPS) to run with,
|
|
how long the test should run, and so on. Almost all options can be configured in the test itself, but as
|
|
you can also pass command line options via the `K6_OPTIONS` variable.
|
|
|
|
For example, you can override the duration of the test with a CLI option:
|
|
|
|
```yaml
|
|
include:
|
|
template: Verify/Load-Performance-Testing.gitlab-ci.yml
|
|
|
|
load_performance:
|
|
variables:
|
|
K6_TEST_FILE: <PATH TO K6 TEST FILE IN PROJECT>
|
|
K6_OPTIONS: '--duration 30s'
|
|
```
|
|
|
|
GitLab only displays the key performance metrics in the MR widget if k6's results are saved
|
|
via [summary export](https://k6.io/docs/results-visualization/json#summary-export)
|
|
as a [Load Performance report artifact](../../../ci/pipelines/job_artifacts.md#artifactsreportsload_performance).
|
|
The latest Load Performance artifact available is always used, using the
|
|
summary values from the test.
|
|
|
|
If [GitLab Pages](../pages/index.md) is enabled, you can view the report directly in your browser.
|
|
|
|
### Load Performance testing in Review Apps
|
|
|
|
The CI/CD YAML configuration example above works for testing against static environments,
|
|
but it can be extended to work with [review apps](../../../ci/review_apps) or
|
|
[dynamic environments](../../../ci/environments) with a few extra steps.
|
|
|
|
The best approach is to capture the dynamic URL in a [`.env` file](https://docs.docker.com/compose/env-file/)
|
|
as a job artifact to be shared, then use a custom environment variable we've provided named `K6_DOCKER_OPTIONS`
|
|
to configure the k6 Docker container to use the file. With this, k6 can then use any
|
|
environment variables from the `.env` file in scripts using standard JavaScript,
|
|
such as: ``http.get(`${__ENV.ENVIRONMENT_URL`})``.
|
|
|
|
For example:
|
|
|
|
1. In the `review` job:
|
|
1. Capture the dynamic URL and save it into a `.env` file, e.g. `echo "ENVIRONMENT_URL=$CI_ENVIRONMENT_URL" >> review.env`.
|
|
1. Set the `.env` file to be a [job artifact](../../../ci/pipelines/job_artifacts.md#job-artifacts).
|
|
1. In the `load_performance` job:
|
|
1. Set it to depend on the review job, so it inherits the environment file.
|
|
1. Set the `K6_DOCKER_OPTIONS` variable with the [Docker CLI option for environment files](https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file), for example `--env-file review.env`.
|
|
1. Configure the k6 test script to use the environment variable in it's steps.
|
|
|
|
Your `.gitlab-ci.yml` file might be similar to:
|
|
|
|
```yaml
|
|
stages:
|
|
- deploy
|
|
- performance
|
|
|
|
include:
|
|
template: Verify/Load-Performance-Testing.gitlab-ci.yml
|
|
|
|
review:
|
|
stage: deploy
|
|
environment:
|
|
name: review/$CI_COMMIT_REF_NAME
|
|
url: http://$CI_ENVIRONMENT_SLUG.example.com
|
|
script:
|
|
- run_deploy_script
|
|
- echo "ENVIRONMENT_URL=$CI_ENVIRONMENT_URL" >> review.env
|
|
artifacts:
|
|
paths:
|
|
- review.env
|
|
rules:
|
|
- if: '$CI_COMMIT_BRANCH' # Modify to match your pipeline rules, or use `only/except` if needed.
|
|
|
|
load_performance:
|
|
dependencies:
|
|
- review
|
|
variables:
|
|
K6_DOCKER_OPTIONS: '--env-file review.env'
|
|
rules:
|
|
- if: '$CI_COMMIT_BRANCH' # Modify to match your pipeline rules, or use `only/except` if needed.
|
|
```
|