End-to-end Testing

What is end-to-end testing?

End-to-end testing is a strategy used to check whether your application works as expected across the entire software stack and architecture, including integration of all micro-services and components that are supposed to work together.

How do we test GitLab?

We use Omnibus GitLab to build GitLab packages and then we test these packages using the GitLab QA orchestrator tool, which is a black-box testing framework for the API and the UI.

Testing nightly builds

We run scheduled pipeline each night to test nightly builds created by Omnibus. You can find these nightly pipelines at gitlab-org/quality/nightly/pipelines.

Testing staging

We run scheduled pipeline each night to test staging. You can find these nightly pipelines at gitlab-org/quality/staging/pipelines.

Testing code in merge requests

It is possible to run end-to-end tests (eventually being run within a GitLab QA pipeline) for a merge request by triggering the package-and-qa manual action in the test stage, that should be present in a merge request widget (unless the merge request is from a fork).

Manual action that starts end-to-end tests is also available in merge requests in Omnibus GitLab.

Below you can read more about how to use it and how does it work.

How does it work?

Currently, we are using multi-project pipeline-like approach to run QA pipelines.

Developer triggers a manual action, that can be found in CE / EE merge requests. This starts a chain of pipelines in multiple projects.
The script being executed triggers a pipeline in Omnibus GitLab and waits for the resulting status. We call this a status attribution.
GitLab packages are being built in the Omnibus GitLab pipeline. Packages are then pushed to its Container Registry.
When packages are ready, and available in the registry, a final step in the Omnibus GitLab pipeline, triggers a new GitLab QA pipeline. It also waits for a resulting status.
GitLab QA pulls images from the registry, spins-up containers and runs tests against a test environment that has been just orchestrated by the gitlab-qa tool.
The result of the GitLab QA pipeline is being propagated upstream, through Omnibus, back to the CE / EE merge request.

How do I write tests?

In order to write new tests, you first need to learn more about GitLab QA architecture. See the documentation about it.

Once you decided where to put test environment orchestration scenarios and instance-level scenarios, take a look at the GitLab QA README, the GitLab QA orchestrator README, and the already existing instance-level scenarios.

Where can I ask for help?

You can ask question in the #quality channel on Slack (GitLab internal) or you can find an issue you would like to work on in the gitlab-ce issue tracker, the gitlab-ee issue tracker, or the gitlab-qa issue tracker.

4.5 KiB Raw Blame History