4.8 KiB
4.8 KiB
Flaky tests
What's a flaky test?
It's a test that sometimes fails, but if you retry it enough times, it passes, eventually.
Automatic retries and flaky tests detection
On our CI, we use rspec-retry to automatically retry a failing example a few
times (see spec/spec_helper.rb for the precise retries count).
We also use a home-made RspecFlaky::Listener listener which records flaky
examples in a JSON report file on master (retrieve-tests-metadata and update-tests-metadata jobs), and warns when a new flaky example
is detected in any other branch (flaky-examples-check job). In the future, the
flaky-examples-check job will not be allowed to fail.
This was originally implemented in: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13021.
Problems we had in the past at GitLab
rspec-retryis bitting us when some API specs fail: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9825- Sporadic RSpec failures due to
PG::UniqueViolation: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9846 - FFaker generates funky data that tests are not ready to handle (and tests should be predictable so that's bad!):
- Make
spec/mailers/notify_spec.rbmore robust: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10015 - Transient failure in spec/requests/api/commits_spec.rb: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9944
- Replace FFaker factory data with sequences: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10184
- Transient failure in spec/finders/issues_finder_spec.rb: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10404
- Make
Time-sensitive flaky tests
- https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10046
- https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10306
Array order expectation
Feature tests
- Be sure to create all the data the test need before starting exercize: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12059
- Bis: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12604
- Bis: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12664
- Assert against the underlying database state instead of against a page's content: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10934
Capybara viewport size related issues
- Transient failure of spec/features/issues/filtered_search/filter_issues_spec.rb: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10411
Capybara JS driver related issues
- Don't wait for AJAX when no AJAX request is fired: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10454
- Bis: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12626
PhantomJS / WebKit related issues
- Memory is through the roof! (TL;DR: Load images but block images requests!): https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12003
Resources
- Flaky Tests: Are You Sure You Want to Rerun Them?
- How to Deal With and Eliminate Flaky Tests
- Tips on Treating Flakiness in your Rails Test Suite
- 'Flaky' tests: a short story
- Using Insights to Discover Flaky, Slow, and Failed Tests