Flaky Test Detection in CI: Rerun and Quarantine

Flaky test detection in CI: rerun the same commit, read results per step with stresseur ci, then decide whether to quarantine or fix each flake.

Mustafa BayramogluMustafa BayramogluAbstract illustration of an amber line passing through a row of gates, one of which sits slightly out of line

Detect a flaky test by running it more than once on the same commit. A test that passes on one run and fails on another, with no change to the code, is flaky. In CI that means repeating tests on purpose, reading the results per step, and then choosing for each flake whether to fix it or to quarantine it for a short, fixed time. This post covers the detection and that choice.

Rerun the same commit to find a flaky test

One run cannot show a flake in an end-to-end suite. A red result might be a real bug, and a green one might be luck. Two runs of the same commit that disagree are the signal. Google reports a continual rate of about 1.5% of all test runs coming back flaky. It also found that about 84% of the transitions it observed from pass to fail involved a flaky test.

There are three ways to produce the disagreement:

  • Rerun failures. When a test fails, run it again on the same commit. If it passes, record it as flaky.
  • Repeat on purpose. Run a test or a flow many times on a commit that you believe is good. Any failure is a flake.
  • Read the pattern per step. For a multi-step test, note which step fails. A flake tends to sit in one step.

This works for every layer of a suite, browser or API. It tells you that a test is unreliable. It does not tell you why, and the why decides the fix. The usual causes and their fixes are in flaky tests: causes and fixes, so this post does not repeat them.

Rerun-based detection in a browser suite

Playwright shows the pattern in its simplest form. It does not retry failing tests unless you ask. With retries on, it labels a test that failed on the first run and passed when retried as “flaky”. Retries alone can turn a flaky run green, so the label is easy to miss in a long log.

Two flags make detection explicit:

  • --repeat-each <N> runs each test N times. Point it at one suspect test and a high N, such as npx playwright test checkout.spec.ts --repeat-each=20, to see whether it ever disagrees with itself.
  • --fail-on-flaky-tests fails the run if any test is flagged as flaky. It is off by default. Turn it on for the job that detects flakes, so a retry cannot hide one.

Repeating a browser suite many times means running that many browser sessions, so spend repeats on suspects and keep the full-suite repeat for a schedule. For checks that never needed a browser, the side-by-side with Playwright shows what each layer is for and where an API flow is the cheaper place to repeat a check.

Per-step reporting with stresseur ci

For API flows, repetition is built into the command. npx stresseur ci runs every flow several times and reports per step. In the Stresseur workflow that is 13 runs per flow, 1 warm-up and 12 measured, and --warmup and --iterations change the counts.

The command exits 1 if any step failed. A step counts as passed only when it succeeded in every run, warm-ups included. So a step that fails in 2 of 13 runs fails the job. A flake that fails even once in those 13 runs fails the job on the pull request that introduced it, not three merges later.

The report names the step, and the pattern is yours to read:

  • A step that fails in every run is a bug or a broken environment.
  • A step that fails in some runs and passes in others is a flake candidate. Check what it depends on: a fixed name, a shared row, a call to a service you do not control.
  • A flow that is slow but never fails is a different problem. When the CI machine was slow and your code was not, the result is Runner noise, re-run, and it never fails the check.

To see why a step failed, add --capture failed. It keeps the request and response of failed steps, masked. The server’s answer on the failing run is often enough to name the cause.

The pull request check has its own gate. In api-tests/stresseur.yml, error_rate.fail_above makes a failed flow fail the check only when more than the set share of runs failed. Treat that as a tolerance for flakes, and set it deliberately, because every flake you tolerate is one you have decided not to fix yet.

Quarantine or fix

Once a test is known to be flaky, it needs one of two outcomes.

Situation Action
The cause is known and the fix is small Fix it now.
The test covers a critical journey Fix it now, and keep it required.
The cause is unknown and the test blocks merges Quarantine it, with an owner and a deadline.
The deadline passed with no fix Fix it or delete it.

Quarantine takes the test off the critical path; file a bug for it at the same time, so someone owns reducing the flakiness. That keeps the rest of the suite trusted. The cost is that quarantine could easily mask a real race condition or some other bug in the code being tested. So keep a quarantined test running where it cannot block a merge, and watch it. A quarantine that nobody watches is a deletion with extra steps.

Do not make blanket retries your policy. Google has a way to mark a test as flaky so that it reports a failure only if it fails 3 times in a row. That reduces false positives, but encourages developers to ignore flakiness in their own tests. A retry is a detector. Use it to find flakes, and then move each one to fix or quarantine.

Keep detection cheap enough for every pull request

Repeats cost time, so put them where they are cheap. Repeating a flow 13 times is part of the Stresseur workflow, which is why it fits on every pull request. Our guide to running API tests on every pull request shows the workflow. Our post on API smoke tests on every pull request covers keeping the set small. Run targeted browser repeats on the tests you already suspect, and a full-suite repeat on a schedule.

Conclusion

Run the same commit more than once and treat disagreement as the finding. Use --repeat-each and --fail-on-flaky-tests in the browser suite, and read stresseur ci per step for API flows. Then give every flake one of two outcomes: a fix, or a quarantine with an owner and a deadline. Retries find flakes. They are not a place to leave them.

Frequently asked questions

How many times should I rerun a test to detect flakiness?

There is no universal number. Playwright runs each test once unless you pass --repeat-each. The Stresseur CI workflow runs each flow 13 times, 1 warm-up and 12 measured. Pick a count your pipeline can afford on every pull request, and raise it for a test you already suspect.

Is a test that passes on retry flaky?

Usually yes. Playwright's own category for a test that failed on the first run and passed when retried is "flaky". The retry shows you the problem; it does not remove it.

Does a slow CI runner mean a test is flaky?

No. In Stresseur a flow that is slow because the CI machine was slow shows as Runner noise, re-run, and never fails the check. Re-run the job.

Should I delete a flaky test?

Delete it only when nobody will fix it. A quarantined test leaves the critical path and gets a bug filed for it, so the quarantine needs an owner and a deadline. When the deadline passes with no fix, a test that gives no reliable signal costs more than it returns.

All posts