= How To Manage Flaky Tests in your CI Workflows
// tag::header[]
:author: Li Haoyi
:revdate: 1 January 2025

_{author}, {revdate}_



Many projects suffer from the problem of flaky tests: tests that pass or fail
non-deterministically. These cause confusion, slow development cycles, and endless
arguments between individuals and teams in an organization.

This article dives deep into working with flaky tests, from the perspective of someone
who built the first flaky test management systems at Dropbox and Databricks and maintained
the related build and CI workflows over the past decade. The issue of flaky tests can be
surprisingly unintuitive, with many "obvious" approaches being ineffective
or counterproductive. But it turns out there _are_ right and wrong answers to many of
these issues, and we will discuss both so you can better understand what managing flaky tests
is all about.

// end::header[]

== What Causes Flaky Tests?

A flaky test is a test that sometimes passes and sometimes fails, non-deterministically.

Flaky tests can be caused by a whole range of issues:

=== Race conditions within tests

Often this manifest as `sleep`/`time.sleep`/`Thread.sleep` calls in your test
expecting some concurrent code path to complete, which may or may not wait
long enough depending on how much CPU contention there is slowing down your test code.
But any multi-threaded or multi-process code has the potential for race conditions or
concurrency bugs, and these days most systems make use of multiple cores.

=== Race conditions between tests

Two tests both reading and writing
to the same global variables or files (e.g. in `~/.cache`) causing non-deterministic
outcomes. These can be tricky, because every test may pass when run alone, and the
test that fails when run in parallel may not be the one that is misbehaving!

=== Test ordering dependencies

Even in the absence of parallelism, tests may interfere
with one another by mutating global variables or files on disk. Depending on the exact
tests you run or the order in which you run them, the tests can behave differently
and pass or fail unpredictably. Perhaps not strictly non-deterministic - the same
tests run in the same order will behave the same - but practically non-deterministic
since different CI runs may run tests in different orders.
xref:3-selective-testing.adoc[Selective Testing] may cause this kind of issue,
or dynamic load-balancing of tests between parallel workers to minimize total wall
clock time (which https://github.com/dropbox/changes[Dropbox's Changes CI system] did)

=== Resource contention

Depending on exactly how your tests are run, they may rely
on process memory, disk space, file descriptors, ports, or other limited
physical resources. These are subject to noisy neighbour problems, e.g. an
overloaded linux system with too many processes using memory will start OOM-Killing
processes at random.

=== External flakiness

Integration or end-to-end tests often
interact with third-party services, and it is not uncommon for the service
to be flaky, rate limit you, have transient networking errors, or just be entirely
down for some period of time. Even fundamental workflows like "downloading
packages from a package repository" can be subject to flaky failures

Sometimes the flakiness is test-specific, sometimes it is actually flakiness in the
code being tested, which may manifest as real flakiness when customers are trying
to use your software. Both scenarios manifest the same to developers - a test passing
and failing non-deterministically when run locally or in CI.

== Why Are Flaky Tests Problematic?

=== Development Slowdown

Flaky tests generally make it impossible to know the state of your test suite,
which in turns makes it impossible to know whether the software you are working on
is broken or not, which is the reason you wanted tests in the first place.
Even a small number of flaky tests is enough to destroy the core value of your test suite.

* Ideally, if the tests pass on a proposed code change, even someone
  unfamiliar with the codebase can be confident that the code change did not break
  anything.

* Once test failures start happening spuriously, it quickly becomes
  impossible to get a fully "green" test run without failures

* So in order to make
  validate a code change (merging a pull-request, deploying a service, etc.) the developer then
  needs to individually triage and make judgements on those test failures to determine
  if they are real issues or spurious

This means that you are back to the "manual" workflow of developers squinting at
test failures to decide if they are relevant or not. This risks both false positives
and false negatives:

* A developer may waste time triaging a failure that turns out to be a flake, and
  unrelated to the change being tested

* A developer may wrongly judge that a test failure is spurious, only for it to be a
  real failure that causes breakage for customers once released.

Even relatively low rates of flakiness can cause issues. For example, consider the following:

* A codebase with 10,000 tests
* 1% of the tests are flaky
* Each flaky test case fails spuriously 1% of the time

Although 1% of tests each failing 1% of the time may not seem like a huge deal, it means
that someone running the entire test suite only has a `0.99^100 = ~37%` chance of getting
a green test report! The other 63% of the time, someone running the test suite without any
real breakages gets one or more spurious failures, that they then have to spend time and energy
triaging and investigating. If the developer needs to retry the test runs to get a successful
result, they would need to retry on average `1 / 0.37 = 2.7` times: in this scenario
the retries alone may be enough to increase your testing latencies and infrastructure costs by
170%, on top of the manual work needed to triage and investigate the test failures!


=== Inter-team Conflict

One fundamental issue with flaky tests is organizational:

* The team that owns a test benefits from the test running and providing coverage
  for the code they care about

* Other teams that run the test in CI suffer from the spurious failures and wasted time
  that hitting flaky tests entails

xref:3-selective-testing.adoc[Selective Testing] can help mitigate this to some extent by
letting you avoid running unrelated tests, but it doesn't make the problem fully disappear.
For example, a downstream service may be triggered every time an upstream utility library
is changed, and if the tests are flaky and the service and library are owned by different
teams, you end up with the conflict described above.

What ends up happening is that _nobody_ prioritizes fixing their flaky tests, because
that is the selfishly-optimal thing to do, but as a result _everyone_
suffers from _everyone else's_ flaky tests, even if everyone would be better if all flaky tests
were fixed. This is a classic https://en.wikipedia.org/wiki/Tragedy_of_the_commons[Tragedy of the Commons],
and as long as flaky tests are allowed to exist, this will result in
endless debates or arguments between teams about who needs to fix their flaky tests,
wasting enormous amounts of time and energy.

== Mitigating Flaky Tests

In general it is impossible to completely avoid flaky tests, but you can take steps to
mitigate them:

1. Avoid race conditions in your application code to prevent random crashes or behavioral changes
   affecting users, and avoid race conditions in your test code

2. Run parallel test processes inside "sandbox" empty temp folders, to try and avoid
   them reading and writing to the same files on the filesystem and risking race conditions.
   (See xref:mill:ROOT:depth/sandboxing.adoc[Mill Sandboxing])

3. Run test processes inside CGroups to mitigate resource contention: e.g. if every test process is limited
   in how much memory it uses, it cannot cause memory pressure that might cause other tests
   to be OOM-killed (See Bazel's https://github.com/bazelbuild/bazel/pull/21322[Extended CGroup Support],
   which we implemented in https://www.databricks.com/blog/2021/10/14/developing-databricks-runbot-ci-solution.html[Databricks' Runbot CI system])

4. Mock out external services: e.g. AWS and Azure can be mocked using https://www.localstack.cloud/[LocalStack], parts of Azure
   Kubernetes can be mocked using https://kind.sigs.k8s.io/[KIND], etc..

5. xref:3-selective-testing.adoc[Selective Testing], e.g. via
   xref:mill::large/selective-execution.adoc[Mill's Selective Test Execution], reduces the
   number of tests you run
   and thus  the impact of flakiness,

However, although you can mitigate the flakiness, you should not expect to make it go away
entirely.

* Race conditions _will_ find their way into your code despite your best efforts, and you _will_
  need some hardcoded timeouts to prevent your test suite hanging forever.

* There will always be _some_ limited physical resource you didn't realize could run out,
  until it does.

* Mocking out third-party services never ends up working 100%: inevitably
  you hit cases where the mock isn't accurate enough, or trustworthy enough, and you still
  need to test against the real service to get confidence in the correctness of your system.

End-to-end tests and integration tests are especially prone to flakiness, as are UI
tests exercising web or mobile app UIs.

As a developer, you should work hard in trying to make your application and test
code as deterministic as possible. You should have a properly-shaped
https://martinfowler.com/articles/practical-test-pyramid.html[Test Pyramid], with more small unit
tests that tend to be stable and fewer integration/end-to-end/UI tests that tend to be flaky.
But you should also accept that despite your best efforts, flaky tests _will_ appear, and so you
will need some plan or strategy to deal with them when they do.

== How Not To Manage Flaky Tests

Flaky test management can be surprisingly counter-intuitive. Below we discuss some common
mistakes people make when they first start dealing with flaky tests.

[#_do_not_block_code_changes_on_flaky_tests]
=== Do Not Block Code Changes on Flaky Tests

The most important thing to take note of is that you should not block
code changes on flaky tests: merging pull-requests, deploying services, etc.

That is despite blocking code changes being the default and most obvious behavior: e.g.
if you wait for a fully-green test run before merging a code change, and a flaky test
makes the test run red, then it blocks the merge.However, this is not a good workflow
for a variety of reasons:

1. A flaky failure when testing a code change does not indicate the code change caused
that breakage.So blocking the merge on the flaky failure just prevents progress
without actually helping increase system quality.

2. The flaky test may be in a part of the system totally unrelated to the code change
   being tested, which means the individual working on the code change has zero context
   on why it might be flaky, and unexpectedly context switching to deal with the flaky test
   is mentally costly.

3. Blocking progress on a flaky test introduces an incentives problem: The code/test owner
   benefits from the flaky test's existence, but other people working in that codebase
   get blocked with no benefit. This directly leads to the endless xref:_inter_team_conflict[]
   mentioned earlier.

Although _"all tests should pass before merging"_ is a common requirement, it is ultimately
unhelpful when you are dealing with flaky tests.

=== Preventing Flaky Tests From Being Introduced Is Hard

It can be tempting to try and "https://en.wikipedia.org/wiki/Shift-left_testing[Shift Left]"
your flaky test management, to try and catch them before they end up landing in your codebase.
But doing so ends up being surprisingly difficult.

Consider the example we used earlier: 10,000 tests, with 1% of them flaky, each failing 1% of
the time. These are arbitrary numbers but pretty representative of what you will likely find
in the wild

1. If someone adds a new test case, in order to have a 95% confidence that it is not flaky,
  you would need to run it about 300 times (`log(0.05) / log(0.99)`).

2. Even if we do run every new test 300 times, that 1 in 20 flaky tests will still slip through,
  and over time will still build up into a population of flaky tests actively causing flakiness
  in your test suite

3. Furthermore, many tests are not flaky alone! Running the same test 300 times in
  isolation may not demonstrate any flakiness, since e.g. the test may only be flaky when
  run in parallel with another test due to <<Race conditions between tests>> or <<Resource contention>>,
  or in a specific order after other tests due to <<Test ordering dependencies>>.

4. Lastly, it is not only new tests that are flaky! When I was working on this area at Dropbox
  and Databricks, the majority of flaky tests we detected were existing tests that
  were stable for days/weeks/months before turning flaky (presumably due to a code change
  in the application code or test code). Blocking new tests that are flaky does nothing
  to prevent the code changes causing old tests to become flaky!

To block code changes that cause either new and old tests from becoming flaky, we would need
to run every single test about 300 times on each pull request, to give us 95% confidence that
each 1% flaky test introduced by the code change would get caught. This is prohibitively
slow and expensive, causing a test suite that may take 5 minutes to run costing $1 to instead
take 25 hours to run costing $300.

In general, it is very hard to block flaky tests "up front". You have to accept that
over time some parts of your test suite will become flaky, and then make plans on how
to respond and manage those flaky tests when they inevitably appear.

== Managing Flaky Tests

Once flaky tests start appearing in your test suite, you need to do something about them.
This generally involves (a) noticing that flaky tests exist, (b) identifying which tests
are flaky, and (c) mitigating those specific problematic test to prevent them from
causing pain to your developers.

=== Monitor Flaky Tests Asynchronously

As mentioned earlier, <<Preventing Flaky Tests From Being Introduced Is Hard>>.
Thus, you must assume that flaky tests _will_ make their way into your test suite,
and monitor the flakiness when it occurs. This can be done in a variety of ways, for example:

1. Most CI systems allow manual retries, and developers usually retry tests they suspect are
   flaky. If a test fails once then passes when retried on the same version of the code, it
   was a flaky failure. This is the metric we used in
   https://www.databricks.com/blog/2021/10/14/developing-databricks-runbot-ci-solution.html[Databricks' CI system]
   to monitor the flaky test numbers.

2. Some CI systems or test frameworks have automatic retries: e.g. in https://github.com/dropbox/changes[Dropbox's Changes CI system]
   all tests were retried twice by default. If a test fails initially and then
   passes on the retry, it is flaky: the fact that it's non-deterministic means that
   next time, it might fail initially and then fail on the retry!

3. Most CI systems run tests to validate code changes before merging, and then run tests
   again to validate the code post-merge. Post-merge should "always" be green, but sometimes
   suffers breakages or flakiness. If a test passes, fails, then passes on three consecutive
   commit test runs post-merge, it is likely to be flaky. Breakages
   tend to cause a string of consecutive test failures before being fixed or reverted, and
   very rarely get noticed and dealt with immediately

Notably, most test failures when validating code changes (e.g. on pull requests) are not useful
here: tests are _meant_ to break when validating code changes in order to catch problems!
Hence the need for the slightly-roundabout ways above to determine what tests are flaky,
by looking for failures at times when you wouldn't expect failures to occur.

Once you have noticed a test is flaky, there are two main options: retries and quarantine

=== Retrying Flaky Tests

Retries are always controversial. A common criticism is that they can mask real flakiness
in the system that can cause real problems to customers, which is true. However,
we already discussed why we xref:_do_not_block_code_changes_on_flaky_tests[should
not block code changes on flaky tests], since doing so just causes pain while
not being an effective way of getting the flakiness fixed.

Furthermore, developers
are going to be manually retrying flaky tests anyway: whether by restarting the job
validating their pull request, or running the test manually on their laptop or devbox
to check if it's truly broken. Thus, we should feel free to add automatic retries around
flaky tests to automate that tedious manual process.

Retrying flaky tests can be surprisingly effective. As mentioned earlier, even
infrequently flaky tests can cause issues, with a small subset of tests flaking
1% of the time being enough to block all progress. However, one retry
turns it into a 0.01% flaky test, and two retries turns it into a 0.0001% flaky test.
So even one or two retries is enough to make most flaky tests stable enough to not cause issues.

Retrying flaky tests has two weaknesses:

==== Retries can be expensive for real failures

If you retry a test twice, that
means that an actually-failed test would run three times before giving up.
If you retry every test by default, and a code change breaks a large number of
them, running all those failing tests three times can be a significant performance
and latency penalty

To mitigate this, you should generally avoid "blanket" retries, and only add
retries around specific tests that you have detected as being flaky

==== Retries may not work if not coarse grained enough

For example, if `test_a` fails
due to interference with `test_b` running concurrently, retrying `test_a`
immediately while `test_b` is still running will fail again. Or if the flakiness is
due to some bad state on the filesystem, the test may continue flaking until
it is run on a completely new machine with a clean filesystem.

This failure mode can be mitigated by retrying the failed tests only after the
entire test suite has completed, possibly on a clean test machine.

=== Auto-Quarantining Flaky Tests

Quarantine involves detecting that a test is flaky, and simply not counting it when deciding
whether or not to accept a code change for merge or deployment.

This is much more aggressive than retrying flaky tests, as even real breakages will get
ignored for quarantined tests. You effectively lose the test coverage given by a particular
test for the period while it is quarantined. Only when someone eventually fixes the flaky test
can it be removed from quarantine and can begin running and blocking code changes again.

Quarantining is best automated, both to remove busy-work of finding/quarantining
flaky tests, and to avoid the inevitable back-and-forth between the people
quarantining the tests and the people whose tests are getting quarantined.


==== Why Quarantine?

The obvious value of quarantining flaky tests is that it unblocks merging of code changes
by ignoring flaky tests that are probably not relevant. Quarantime basically automates what
people do manually in the presence of flaky tests anyway:

* When enough tests are flaky, eventually developers are going to start merging/deploying code
  changes despite the failures being present, because getting a "fully green" test run is
  impossible

* When that happens, the developer is not going to be able to tell whether the failure
  is flaky or real, so if a code change causes a real breakage in that test the
  developer is likely going to not notice and merge/deploy it anyway!

So although naively it seems like quarantining flaky tests cost you test coverage, in
reality it costs you nothing and simply automates the loss of coverage that you are going
to suffer anyway. It simply saves a lot of manual effort in having your developers manually
deciding which test failures to ignore based on what tests they remember to be flaky, since
now the quarantine system remembers the flaky tests and ignores them on your behalf.

==== Why Quarantine? Part 2

The non-obvious value of quarantining flaky tests is that it aligns incentives across a
development team or organization:

* Normally, a flaky test meant the test owner continues to benefit from the test coverage while
  other teams suffered from the flakiness

* With auto-quarantine, a flaky test means the test owner both benefits from the test coverage for
  health tests and suffers the lack of coverage caused by their flaky test being quarantined.

This aligning of incentives means that with auto-quarantine enabled, the normal
endless discussions and disputes about flaky tests tend to disappear. The test owner
can decide themselves how urgently they need to fix a quarantined flaky test, depending
on how crucial that test coverage is, or even if they should fix it at all! Other teams
are not affected by the quarantined flaky test, and do not care what the test owner ends
up deciding

Most commonly, quarantining is automatic, while un-quarantining a test can be automatic or manual.
Due to the non-determinstic nature of flakiness, it's often hard to determine whether a flaky
test has been truly fixed or not, but it turns out it doesn't matter. If you try to fix a test,
take it out of quarantine, and it turns out to be still flaky, the auto-quarantine system will
just put it back into quarantine for you to take another look at it.

== Implementing Flaky Test Management Systems

So far, all the discussion in this article has been at a high level. Exactly how to implement
it is left as an exercise to the reader, but is usually a mix of:

* `retry{}` helpers in multiple languages you can sprinkle through your test code where necessary
* A SQL database storing historical test results and retries
* A SQL database or a text file committed in-repo to track quarantined tests
* A service that looks at historical test results and retries and decides when/if to quarantine a test
* Tweaks to your existing CI system to be able to work with all of the above: ignoring quarantined
  tests, tracking retry counts, tracking test results, etc.
* Some kind of web interface giving you visibility into all the components and workflows above,
  so when things inevitably go wrong you are able to figure out what's misbehaving

Usually flaky test management starts off as an entirely manual process, which works fine for small
projects. But as the size of the project grows, you inevitably need to augment the manual work
with some basic automation, and over time build out a fully automated system to do what you want.
So far I have not seen a popular out-of-the-box solution for this, and in my interviews with ~30
silicon valley companies it seems everyone ends up building their own. The
https://github.com/dropbox/changes[Dropbox CI System] and
https://www.databricks.com/blog/2021/10/14/developing-databricks-runbot-ci-solution.html[Databricks CI System]
I worked on both had their flaky test management bespoke and built in to the infrastructure.

None of the techniques discussed in this article are rocket science, and the challenge is mostly
just plumbing the necessary data back and forth between different parts of your CI system. But
hopefully this high-level discussion of how to manage flaky tests should give you a head start,
and save you the weeks or months it would take to learn the same
things that I have learned working on flaky tests over the past decade.
