GitLab tests in the Continuous Integration (CI) context

Test suite parallelization on the CI

Our current CI parallelization setup is as follows:

The knapsack job in the prepare stage that is supposed to ensure we have a knapsack/${CI_PROJECT_NAME}/rspec_report-master.json file:
- The knapsack/${CI_PROJECT_NAME}/rspec_report-master.json file is fetched from S3, if it's not here we initialize the file with {}.
Each rspec x y job are run with knapsack rspec and should have an evenly distributed share of tests:
- It works because the jobs have access to the knapsack/${CI_PROJECT_NAME}/rspec_report-master.json since the "artifacts from all previous stages are passed by default".
- the jobs set their own report path to KNAPSACK_REPORT_PATH=knapsack/${CI_PROJECT_NAME}/${JOB_NAME[0]}_node_${CI_NODE_INDEX}_${CI_NODE_TOTAL}_report.json.
- if knapsack is doing its job, test files that are run should be listed under Report specs, not under Leftover specs.
The update-knapsack job takes all the knapsack/${CI_PROJECT_NAME}/${JOB_NAME[0]}_node_${CI_NODE_INDEX}_${CI_NODE_TOTAL}_report.json files from the rspec x y jobs and merge them all together into a single knapsack/${CI_PROJECT_NAME}/rspec_report-master.json file that is then uploaded to S3.

After that, the next pipeline will use the up-to-date knapsack/${CI_PROJECT_NAME}/rspec_report-master.json file.

The GitLab test suite is monitored for the master branch, and any branch that includes rspec-profile in their name.

A public dashboard is available for everyone to see. Feel free to look at the slowest test files and try to improve them.

Rails logging to log/test.log is disabled by default in CI for performance reasons. To override this setting, provide the RAILS_ENABLE_TEST_LOG environment variable.