2019-07-07 11:18:12 +05:30
---
2020-06-23 00:09:42 +05:30
stage: Verify
group: Testing
2021-02-22 17:27:13 +05:30
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
2021-03-08 18:12:59 +05:30
author: Vincent Tunru
author_gitlab: Vinnl
2019-09-04 21:01:54 +05:30
type: tutorial
2021-03-08 18:12:59 +05:30
date: 2019-02-18
description: 'Confidence checking your entire app every time a new feature is added can quickly become repetitive. Learn how to automate it with GitLab CI/CD.'
2019-07-07 11:18:12 +05:30
---
2021-03-08 18:12:59 +05:30
<!-- vale off -->
2019-07-07 11:18:12 +05:30
# End-to-end testing with GitLab CI/CD and WebdriverIO
[Review Apps ](../../review_apps/index.md ) are great: for every merge request
(or branch, for that matter), the new code can be copied and deployed to a fresh production-like live
2021-01-03 14:25:43 +05:30
environment, reducing the effort to assess the impact of changes. Thus, when we use a dependency manager like
2019-07-07 11:18:12 +05:30
[Dependencies.io ](https://www.dependencies.io/ ), it can submit a merge request with an updated dependency,
and it will immediately be clear that the application can still be properly built and deployed. After all, you can _see_ it
running!
2021-01-29 00:20:46 +05:30
< img src = "img/deployed_dependency_update.png" alt = "dependencies.io" >
2019-07-07 11:18:12 +05:30
However, looking at the freshly deployed code to check whether it still looks and behaves as
expected is repetitive manual work, which means it is a prime candidate for automation. This is
where automated [end-to-end testing ](https://martinfowler.com/bliki/BroadStackTest.html ) comes in:
having the computer run through a few simple scenarios that requires the proper functioning of all
2019-09-04 21:01:54 +05:30
layers of your application, from the frontend to the database.
In this article, we will discuss how
2019-07-07 11:18:12 +05:30
to write such end-to-end tests, and how to set up GitLab CI/CD to automatically run these tests
against your new code, on a branch-by-branch basis. For the scope of this article, we will walk you
2019-12-21 20:55:43 +05:30
through the process of setting up GitLab CI/CD for end-to-end testing JavaScript-based applications
2019-07-07 11:18:12 +05:30
with WebdriverIO, but the general strategy should carry over to other languages.
We assume you are familiar with GitLab, [GitLab CI/CD ](../../README.md ), [Review Apps ](../../review_apps/index.md ), and running your app locally, e.g., on `localhost:8000` .
2019-09-30 21:07:59 +05:30
## What to test
2019-07-07 11:18:12 +05:30
In the widely-used [testing pyramid strategy ](https://martinfowler.com/bliki/TestPyramid.html ), end-to-end tests act more like a
safeguard: [most of your code should be covered by
unit tests](https://vincenttunru.com/100-percent-coverage/) that allow you to easily identify the source of a problem, should one occur. Rather, you
will likely want to
[limit the number of end-to-end tests ](https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html )
to just enough to give you the confidence that the deployment went as intended, that your
2019-09-30 21:07:59 +05:30
infrastructure is up and running, and that your units of code work well together.
2019-07-07 11:18:12 +05:30
2019-09-30 21:07:59 +05:30
## Selenium and WebdriverIO
2019-07-07 11:18:12 +05:30
2020-04-22 19:07:51 +05:30
[Selenium ](https://www.selenium.dev/ ) is a piece of software that can control web browsers, e.g., to make them
2019-07-07 11:18:12 +05:30
visit a specific URL or interact with elements on the page. It can be programmatically controlled
from a variety of programming languages. In this article we're going to be using the
2019-12-21 20:55:43 +05:30
[WebdriverIO ](https://webdriver.io/ ) JavaScript bindings, but the general concept should carry over
2019-07-07 11:18:12 +05:30
pretty well to
2020-04-08 14:13:33 +05:30
[other programming languages supported by Selenium ](https://www.selenium.dev/documentation/en/legacy_docs/selenium_rc/ ).
2019-07-07 11:18:12 +05:30
## Writing tests
You can write tests using
2019-12-21 20:55:43 +05:30
[several testing frameworks supported by WebdriverIO ](https://webdriver.io/guide/testrunner/frameworks.html ).
2019-07-07 11:18:12 +05:30
We will be using [Jasmine ](https://jasmine.github.io/ ) here:
```javascript
describe('A visitor without account', function(){
it('should be able to navigate to the homepage from the 404 page', function(){
browser.url('/page-that-does-not-exist');
expect(browser.getUrl()).toMatch('page-that-does-not-exist');
browser.element('.content a[href="/"]').click();
2019-09-30 21:07:59 +05:30
2019-07-07 11:18:12 +05:30
expect(browser.getUrl()).not.toMatch('page-that-does-not-exist');
});
});
```
The functions `describe` , `it` , and `browser` are provided by WebdriverIO. Let's break them down one by one.
The function `describe` allows you to group related tests. This can be useful if, for example, you want to
run the same initialization commands (using [`beforeEach` ](https://jasmine.github.io/api/2.9/global.html#beforeEach )) for
multiple tests, such as making sure you are logged in.
The function `it` defines an individual test.
2019-12-21 20:55:43 +05:30
[The `browser` object ](https://webdriver.io/guide/testrunner/browserobject.html ) is WebdriverIO's
special sauce. It provides most of [the WebdriverIO API methods ](https://webdriver.io/api.html ) that are the key to
2019-07-07 11:18:12 +05:30
steering the browser. In this case, we can use
2019-12-21 20:55:43 +05:30
[`browser.url` ](https://webdriver.io/api/protocol/url.html ) to visit `/page-that-does-not-exist` to
hit our 404 page. We can then use [`browser.getUrl` ](https://webdriver.io/api/property/getUrl.html )
2019-07-07 11:18:12 +05:30
to verify that the current page is indeed at the location we specified. To interact with the page,
we can simply pass CSS selectors to
2019-12-21 20:55:43 +05:30
[`browser.element` ](https://webdriver.io/api/protocol/element.html ) to get access to elements on the
2019-07-07 11:18:12 +05:30
page and to interact with them - for example, to click on the link back to the home page.
The simple test shown above
can already give us a lot of confidence if it passes: we know our deployment has succeeded, that the
elements are visible on the page and that actual browsers can interact with it, and that routing
works as expected. And all that in just 10 lines with gratuitous whitespace! Add to that succeeding
unit tests and a successfully completed pipeline, and you can be fairly confident that the
dependency upgrade did not break anything without even having to look at your website.
## Running locally
2021-01-03 14:25:43 +05:30
We'll get to running the above test in CI/CD in a moment. When writing tests,
however, it helps if you don't have to wait for your pipelines to succeed to
determine whether they do what you expect them to do. In other words, let's get
it to run locally.
2019-07-07 11:18:12 +05:30
Make sure that your app is running locally. If you use Webpack,
you can use [the Webpack Dev Server WebdriverIO plugin ](https://www.npmjs.com/package/wdio-webpack-dev-server-service )
that automatically starts a development server before executing the tests.
The WebdriverIO documentation has
2019-12-21 20:55:43 +05:30
[an overview of all configuration options ](https://webdriver.io/guide/getstarted/configuration.html ), but the
2019-07-07 11:18:12 +05:30
easiest way to get started is to start with
2019-12-21 20:55:43 +05:30
[WebdriverIO's default configuration ](https://webdriver.io/guide/testrunner/configurationfile.html ), which
2019-07-07 11:18:12 +05:30
provides an overview of all available options. The two options that are going to be most relevant now are the
`specs` option, which is an array of paths to your tests, and the `baseUrl` option, which points to where your app is
running. And finally, we will need to tell WebdriverIO in which browsers we would like to run our
tests. This can be configured through the `capabilities` option, which is an array of browser names (e.g.
`firefox` or `chrome` ). It is recommended to install
[selenium-assistant ](https://googlechromelabs.github.io/selenium-assistant/ ) to detect all installed
browsers:
```javascript
const seleniumAssistant = require('selenium-assistant');
const browsers = seleniumAssistant.getLocalBrowsers();
config.capabilities = browsers.map(browser => ({ browserName: browser.getId() }));
```
But of course, a simple configuration of `config.capabilities = ['firefox']` would work as well.
If you've installed WebdriverIO as a dependency
(`npm install --save-dev webdriverio`), you can add a line to the `scripts` property in your
`package.json` that runs `wdio` with the path to your configuration file as value, e.g.:
```javascript
"confidence-check": "wdio wdio.conf.js",
```
You can then execute the tests using `npm run confidence-check` , after which you will actually see a
new browser window interacting with your app as you specified.
## Configuring GitLab CI/CD
Which brings us to the exciting part: how do we run this in GitLab CI/CD? There are two things we
need to do for this:
2021-01-29 00:20:46 +05:30
1. Set up [CI/CD jobs ](../../yaml/README.md ) that actually have a browser available.
2019-10-12 21:52:04 +05:30
1. Update our WebdriverIO configuration to use those browsers to visit the review apps.
2019-07-07 11:18:12 +05:30
For the scope of this article, we've defined an additional [CI/CD stage ](../../yaml/README.md#stages )
`confidence-check` that is executed _after_ the stage that deploys the review app. It uses the `node:latest` [Docker
2020-03-13 15:44:24 +05:30
image](../../docker/using_docker_images.md). However, WebdriverIO fires up actual browsers
2019-07-07 11:18:12 +05:30
to interact with your application, so we need to install and run them.
Furthermore, WebdriverIO uses Selenium as a common interface to control different browsers,
so we need to install and run Selenium as well. Luckily, the Selenium project provides the Docker images
[standalone-firefox ](https://hub.docker.com/r/selenium/standalone-firefox/ ) and
[standalone-chrome ](https://hub.docker.com/r/selenium/standalone-chrome/ ) that provide just that for
Firefox and Chrome, respectively. (Since Safari and Internet Explorer/Edge are not open source and
not available for Linux, we are unfortunately unable to use those in GitLab CI/CD).
GitLab CI/CD makes it a breeze to link these images to our `confidence-check` jobs using the
`service` property, which makes the Selenium server available under a hostname based on the image
name. Our job configuration then looks something like this:
```yaml
e2e:firefox:
stage: confidence-check
services:
- selenium/standalone-firefox
script:
- npm run confidence-check --host=selenium__standalone-firefox
```
And likewise for Chrome:
```yaml
e2e:chrome:
stage: confidence-check
services:
- selenium/standalone-chrome
script:
- npm run confidence-check --host=selenium__standalone-chrome
```
Now that we have a job to run the end-to-end tests in, we need to tell WebdriverIO how to connect to
the Selenium servers running alongside it. We've already cheated a bit above by
2019-12-21 20:55:43 +05:30
passing the value of the [`host` ](https://webdriver.io/guide/getstarted/configuration.html#host )
2019-07-07 11:18:12 +05:30
option as an argument to `npm run confidence-check` on the command line.
However, we still need to tell WebdriverIO which browser is available for it to use.
[GitLab CI/CD makes
2020-03-13 15:44:24 +05:30
a number of variables available](../../variables/README.md#predefined-environment-variables)
2019-07-07 11:18:12 +05:30
with information about the current CI job. We can use this information to dynamically set
up our WebdriverIO configuration according to the job that is running. More specifically, we can
tell WebdriverIO what browser to execute the test on depending on the name of the currently running
job. We can do so in WebdriverIO's configuration file, which we named `wdio.conf.js` above:
```javascript
if(process.env.CI_JOB_NAME) {
dynamicConfig.capabilities = [
{ browserName: process.env.CI_JOB_NAME === 'e2e:chrome' ? 'chrome' : 'firefox' },
];
}
```
Likewise, we can tell WebdriverIO where the review app is running - in this example's case, it's on
`<branch name>.flockademic.com` :
```javascript
if(process.env.CI_COMMIT_REF_SLUG) {
dynamicConfig.baseUrl = `https://${process.env.CI_COMMIT_REF_SLUG}.flockademic.com` ;
}
```
And we can make sure our local-specific configuration is only used when _not_ running in CI using
`if (!process.env.CI)` . That's basically all the ingredients you need to run your end-to-end tests
on GitLab CI/CD!
To recap, our `.gitlab-ci.yml` configuration file looks something like this:
```yaml
image: node:8.10
stages:
- deploy
- confidence-check
deploy_terraform:
stage: deploy
script:
# Your Review App deployment scripts - for a working example please check https://gitlab.com/Flockademic/Flockademic/blob/5a45f1c2412e93810fab50e2dab8949e2d0633c7/.gitlab-ci.yml#L315
2020-11-24 15:15:51 +05:30
- echo
2019-07-07 11:18:12 +05:30
e2e:firefox:
stage: confidence-check
services:
- selenium/standalone-firefox
script:
- npm run confidence-check --host=selenium__standalone-firefox
e2e:chrome:
stage: confidence-check
services:
- selenium/standalone-chrome
script:
- npm run confidence-check --host=selenium__standalone-chrome
```
## What's next
If you are setting this up for yourself and want to peek at the working configuration of a
production project, see:
- [Flockademic's `wdio.conf.js` ](https://gitlab.com/Flockademic/Flockademic/blob/dev/wdio.conf.js )
- [Flockademic's `.gitlab-ci.yml` ](https://gitlab.com/Flockademic/Flockademic/blob/dev/.gitlab-ci.yml )
- [Flockademic's tests ](https://gitlab.com/Flockademic/Flockademic/tree/dev/__e2e__ )
2019-12-21 20:55:43 +05:30
There's plenty more that WebdriverIO can do. For example, you can configure a [`screenshotPath` ](https://webdriver.io/guide/getstarted/configuration.html#screenshotPath ) to tell WebdriverIO to take
2019-07-07 11:18:12 +05:30
a screenshot when tests are failing. Then tell GitLab CI/CD to store those
[artifacts ](../../yaml/README.md#artifacts ), and you'll be able to see what went
wrong within GitLab.