6.1 KiB
GitLab QA - End-to-end tests for GitLab
This directory contains end-to-end tests for GitLab. It includes the test framework and the tests themselves.
The tests can be found in qa/specs/features (not to be confused with the unit
tests for the test framework, which are in spec/).
It is part of the GitLab QA project.
What is it?
GitLab QA is an end-to-end tests suite for GitLab.
These are black-box and entirely click-driven end-to-end tests you can run against any existing instance.
How does it work?
- When we release a new version of GitLab, we build a Docker images for it.
- Along with GitLab Docker Images we also build and publish GitLab QA images.
- GitLab QA project uses these images to execute end-to-end tests.
Validating GitLab views / partials / selectors in merge requests
We recently added a new CI job that is going to be triggered for every push
event in CE and EE projects. The job is called qa:selectors and it will
verify coupling between page objects implemented as a part of GitLab QA
and corresponding views / partials / selectors in CE / EE.
Whenever qa:selectors job fails in your merge request, you are supposed to
fix page objects. You should also trigger end-to-end tests
using package-and-qa manual action, to test if everything works fine.
How can I use it?
You can use GitLab QA to exercise tests on any live instance! For example, the
following call would login to a local GDK instance and run all specs in
qa/specs/features:
bin/qa Test::Instance::All http://localhost:3000
Note: If you want to run tests requiring SSH against GDK, you will need to modify your GDK setup.
Writing tests
Running specific tests
You can also supply specific tests to run as another parameter. For example, to run the repository-related specs, you can execute:
bin/qa Test::Instance::All http://localhost -- qa/specs/features/browser_ui/3_create/repository
Since the arguments would be passed to rspec, you could use all rspec
options there. For example, passing --backtrace and also line number:
bin/qa Test::Instance::All http://localhost -- qa/specs/features/browser_ui/3_create/merge_request/create_merge_request_spec.rb:6 --backtrace
Note that the separator -- is required; all subsequent options will be
ignored by the QA framework and passed to rspec.
Overriding the authenticated user
Unless told otherwise, the QA tests will run as the default root user seeded
by the GDK.
If you need to authenticate as a different user, you can provide the
GITLAB_USERNAME and GITLAB_PASSWORD environment variables:
GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password bin/qa Test::Instance::All https://gitlab.example.com
If your user doesn't have permission to default sandbox group
gitlab-qa-sandbox, you could also use another sandbox group by giving
GITLAB_SANDBOX_NAME:
GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password GITLAB_SANDBOX_NAME=jsmith-qa-sandbox bin/qa Test::Instance::All https://gitlab.example.com
All supported environment variables are here.
Sending additional cookies
The environment variable QA_COOKIES can be set to send additional cookies
on every request. This is necessary on gitlab.com to direct traffic to the
canary fleet. To do this set QA_COOKIES="gitlab_canary=true".
To set multiple cookies, separate them with the ; character, for example: QA_COOKIES="cookie1=value;cookie2=value2"
Building a Docker image to test
Once you have made changes to the CE/EE repositories, you may want to build a
Docker image to test locally instead of waiting for the gitlab-ce-qa or
gitlab-ee-qa nightly builds. To do that, you can run from this directory:
docker build -t gitlab/gitlab-ce-qa:nightly .
Quarantined tests
Tests can be put in quarantine by assigning :quarantine metadata. This means
they will be skipped unless run with --tag quarantine. This can be used for
tests that are expected to fail while a fix is in progress (similar to how
skip or pending
can be used).
bin/qa Test::Instance::All http://localhost -- --tag quarantine
If quarantine is used with other tags, tests will only be run if they have at
least one of the tags other than quarantine. This is different from how RSpec
tags usually work, where all tags are inclusive.
For example, suppose one test has :smoke and :quarantine metadata, and
another test has :ldap and :quarantine metadata. If the tests are run with
--tag smoke --tag quarantine, only the first test will run. The test with
:ldap will not run even though it also has :quarantine.
Running tests with a feature flag enabled
Tests can be run with with a feature flag enabled by using the command-line
option --enable-feature FEATURE_FLAG. For example, to enable the feature flag
that enforces Gitaly request limits, you would use the command:
bin/qa Test::Instance::All http://localhost --enable-feature gitaly_enforce_requests_limits
This will instruct the QA framework to enable the gitaly_enforce_requests_limits
feature flag (via the API), run
all the tests in the Test::Instance::All scenario, and then disable the
feature flag again.
Note: the QA framework doesn't currently allow you to easily toggle a feature flag during a single test, as you can in unit tests, but that capability is planned.
Note also that the -- separator isn't used because --enable-feature is a QA
framework option, not an rspec option.