debian-mirror-gitlab/doc/development/testing_guide/end_to_end_tests.md

96 lines
4.5 KiB
Markdown
Raw Normal View History

2019-02-15 15:39:39 +05:30
# End-to-end Testing
2018-03-17 18:26:18 +05:30
2019-02-15 15:39:39 +05:30
## What is end-to-end testing?
2018-03-17 18:26:18 +05:30
2019-02-15 15:39:39 +05:30
End-to-end testing is a strategy used to check whether your application works
as expected across the entire software stack and architecture, including
integration of all micro-services and components that are supposed to work
2018-03-17 18:26:18 +05:30
together.
## How do we test GitLab?
We use [Omnibus GitLab][omnibus-gitlab] to build GitLab packages and then we
2019-02-15 15:39:39 +05:30
test these packages using the [GitLab QA orchestrator][gitlab-qa] tool, which is
a black-box testing framework for the API and the UI.
2018-03-17 18:26:18 +05:30
### Testing nightly builds
We run scheduled pipeline each night to test nightly builds created by Omnibus.
2019-02-15 15:39:39 +05:30
You can find these nightly pipelines at [gitlab-org/quality/nightly/pipelines][quality-nightly-pipelines].
### Testing staging
We run scheduled pipeline each night to test staging.
You can find these nightly pipelines at [gitlab-org/quality/staging/pipelines][quality-staging-pipelines].
2018-03-17 18:26:18 +05:30
### Testing code in merge requests
It is possible to run end-to-end tests (eventually being run within a
[GitLab QA pipeline][gitlab-qa-pipelines]) for a merge request by triggering
2019-02-15 15:39:39 +05:30
the `package-and-qa` manual action in the `test` stage, that should be present
in a merge request widget (unless the merge request is from a fork).
2018-03-17 18:26:18 +05:30
Manual action that starts end-to-end tests is also available in merge requests
2019-02-15 15:39:39 +05:30
in [Omnibus GitLab][omnibus-gitlab].
2018-03-17 18:26:18 +05:30
Below you can read more about how to use it and how does it work.
#### How does it work?
Currently, we are using _multi-project pipeline_-like approach to run QA
pipelines.
2019-02-15 15:39:39 +05:30
1. Developer triggers a manual action, that can be found in CE / EE merge
2018-03-17 18:26:18 +05:30
requests. This starts a chain of pipelines in multiple projects.
2019-02-15 15:39:39 +05:30
1. The script being executed triggers a pipeline in [Omnibus GitLab][omnibus-gitlab]
and waits for the resulting status. We call this a _status attribution_.
2018-03-17 18:26:18 +05:30
2019-02-15 15:39:39 +05:30
1. GitLab packages are being built in the [Omnibus GitLab][omnibus-gitlab]
pipeline. Packages are then pushed to its Container Registry.
2018-03-17 18:26:18 +05:30
1. When packages are ready, and available in the registry, a final step in the
2019-02-15 15:39:39 +05:30
[Omnibus GitLab][omnibus-gitlab] pipeline, triggers a new
[GitLab QA pipeline][gitlab-qa-pipelines]. It also waits for a resulting status.
2018-03-17 18:26:18 +05:30
1. GitLab QA pulls images from the registry, spins-up containers and runs tests
against a test environment that has been just orchestrated by the `gitlab-qa`
tool.
2019-02-15 15:39:39 +05:30
1. The result of the [GitLab QA pipeline][gitlab-qa-pipelines] is being
propagated upstream, through Omnibus, back to the CE / EE merge request.
2018-03-17 18:26:18 +05:30
#### How do I write tests?
In order to write new tests, you first need to learn more about GitLab QA
2019-02-15 15:39:39 +05:30
architecture. See the [documentation about it][gitlab-qa-architecture].
2018-03-17 18:26:18 +05:30
2019-02-15 15:39:39 +05:30
Once you decided where to put [test environment orchestration scenarios] and
[instance-level scenarios], take a look at the [GitLab QA README][instance-qa-readme],
the [GitLab QA orchestrator README][gitlab-qa-readme], and [the already existing
instance-level scenarios][instance-level scenarios].
2018-03-17 18:26:18 +05:30
## Where can I ask for help?
2018-10-15 14:42:47 +05:30
You can ask question in the `#quality` channel on Slack (GitLab internal) or
you can find an issue you would like to work on in
2019-02-15 15:39:39 +05:30
[the `gitlab-ce` issue tracker][gitlab-ce-issues],
[the `gitlab-ee` issue tracker][gitlab-ce-issues], or
[the `gitlab-qa` issue tracker][gitlab-qa-issues].
2018-03-17 18:26:18 +05:30
[omnibus-gitlab]: https://gitlab.com/gitlab-org/omnibus-gitlab
[gitlab-qa]: https://gitlab.com/gitlab-org/gitlab-qa
2019-02-15 15:39:39 +05:30
[gitlab-qa-readme]: https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md
2018-03-17 18:26:18 +05:30
[gitlab-qa-pipelines]: https://gitlab.com/gitlab-org/gitlab-qa/pipelines
2019-02-15 15:39:39 +05:30
[quality-nightly-pipelines]: https://gitlab.com/gitlab-org/quality/nightly/pipelines
[quality-staging-pipelines]: https://gitlab.com/gitlab-org/quality/staging/pipelines
2018-03-17 18:26:18 +05:30
[gitlab-qa-architecture]: https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md
2019-02-15 15:39:39 +05:30
[gitlab-qa-issues]: https://gitlab.com/gitlab-org/gitlab-qa/issues?label_name%5B%5D=new+scenario
[gitlab-ce-issues]: https://gitlab.com/gitlab-org/gitlab-ce/issues?label_name[]=QA&label_name[]=test
[gitlab-ee-issues]: https://gitlab.com/gitlab-org/gitlab-ee/issues?label_name[]=QA&label_name[]=test
[test environment orchestration scenarios]: https://gitlab.com/gitlab-org/gitlab-qa/tree/master/lib/gitlab/qa/scenario
[instance-level scenarios]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/specs/features
[Page objects documentation]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/page/README.md
2018-03-17 18:26:18 +05:30
[instance-qa-readme]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/README.md
[instance-qa-examples]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa