Flaky Tests: Causes and Fixes Below the Browser

A flaky test passes on one run and fails on the next with no change to the code. Four common causes are timing, selectors, test data and shared state. Fix each at its cause. Then ask which of your browser checks need a browser at all, because the ones that do not are the cheapest to make stable.

What makes a test flaky

A flaky test is one that passes or fails inconsistently without any change in the underlying code. In an end-to-end suite the causes usually include concurrency issues, timing dependencies, reliance on external systems and a lack of isolation between tests. An analysis that Trunk summarizes in its guide to flaky tests found the most common causes to be async wait (45%), concurrency (20%) and test order.

Playwright gives the symptom a name. A test that failed on the first run and passed when retried gets the label “flaky”. That label tells you a test is unreliable. It does not tell you why, and the why decides the fix.

This post covers browser tests and API flow tests, the two layers described in our end-to-end testing guide. The four causes below come up often in browser tests. The section on moving checks down a layer comes after them.

Timing: waiting for the wrong thing

Timing flakes happen when a test acts before the app is ready. The usual patch is a pause, and a fixed sleep waits too long or not long enough. A slow run wastes time. A fast run still fails when the page takes longer than the sleep.

Wait for a condition instead. In Playwright, web first assertions wait until the expected condition is met. If a message takes half a second to appear, an assertion such as toBeVisible() waits and retries, so the test needs no sleep. Waiting on a condition removes the guesswork that a sleep adds.

Some variance remains. Animations, slow third-party scripts and network jitter still add noise to a browser test. You cannot wait that out, but you can stop depending on it. Playwright’s advice is to only test what you control, and not to test servers you do not control. Stub the third-party call, or move the check that needs it to a place where a slow script cannot reach it.

Selectors: tests tied to markup

A browser test has to find elements, and how it finds them decides how often a harmless change breaks it. Playwright’s best practices say to avoid relying on implementation details such as the CSS class of an element. A class name exists for styling. Nobody promised the test that it would stay put.

On one small shop app we renamed the Edit button’s CSS class, a two-line change in the front end that left the API alone. The UI test timed out after 30 s and failed, while the API flow still passed all 8 of its steps. The UI test was partly at fault, since it located the button by CSS class, the pattern Playwright’s docs warn against. A role-based locator would have survived the rename.

That is the first hint about layers. A browser test can break on markup because it sees markup. An API flow never sees the page, so a UI-only change cannot break it. That holds for UI-only changes. When the API itself changes, a flow fails like any other test.

Test data: the second run fails

A test that passes once and fails on the second run has a data problem. If a sign-up test always registers the same email and emails must be unique, it works on a clean database and fails once that email is taken. Every run needs fresh data, or a reset before it starts.

Give every run its own values. Build names and emails from a run ID or a random suffix, and have the test create what it needs instead of assuming a row exists. Check the outcome the user cares about, such as the saved order, and do not stop at a page that loaded.

This applies to both layers. Stresseur’s generated flows send fresh sign-up values each run and check each answer, and go beyond the status code. Since 0.4.0 they also read a created item back: after a create, a later read of the list or page must hold the new id, or a value the user typed, made unique per run. Fresh data and a read-back catch the failure that a bare status check hides.

Shared state: order changes the result

When tests lean on the same login, cart or database row, the result depends on which test runs first. The suite can pass in one order and fail in another, and the test that fails is often not the test that caused the problem.

Playwright’s guidance is direct: each test should be completely isolated from another test and should run independently with its own local storage, session storage, data and cookies. That is a good rule for any layer. Create the user inside the test. Log in inside the test. Clean up what the test made, or make what it made unique so that cleanup does not matter.

If a test fails only after another one, run the pair alone and swap their order. A test that depends on a leftover is easy to find that way and cheap to fix. Fix it before you reach for a retry, because a retry hides a flake; it doesn’t remove it.

Which checks to move below the browser

The four causes above all get worse in the browser, where a page and its scripts sit between your test and the thing you want to check. Many checks do not need any of that. If the browser adds nothing to a check, run it at the API layer. If the check is about what the user’s screen shows, keep it in the browser.

Move these to API flows:

• Business journeys. Checkout math, permissions and state changes across several requests.
• Edge cases. Bad input, missing fields, limits and expired tokens.
• Checks you want on every pull request. They run in seconds, so nobody skips them.

Keep these in the browser:

• Layout. Does the button show, and is the form usable on a phone.
• Client-side logic. Form checks, state and routing that stay inside the page and never call the API.
• Accessibility and cross-browser behavior.
• Third-party widgets that run inside the page, such as an embedded payment form.

Layout, client-side logic and accessibility are visible only to a browser test. Playwright is the browser tool we recommend for that layer. It can also send requests straight to your REST API with no browser, so one tool can cover both layers if you are happy writing the API steps by hand. Stresseur takes the other route: it records how an API is really used during development, turns those flows into YAML tests, and runs them in CI on every pull request.

Many teams pair a few browser smoke tests, one for each critical journey, with API flows that cover the other journeys and their edge cases. Our guide to what to automate first covers which API checks come first, and the multi-step flows guide shows how tokens and IDs pass between steps.

The same journey, two layers

The full method and setup are in the end-to-end testing guide. In short, one journey on a small shop app ran as a Playwright test and as a recorded Stresseur flow.

	Browser test	API flow
10 runs, total time	11.2 s	2.2 s
Passed, of 50 runs	50 of 50	50 of 50
Button CSS class renamed	Failed	Passed

Ten runs took 11.2 s in the browser and 2.2 s as a flow, so the flow was faster to run. Renaming the button’s CSS class failed the browser test and left the flow green, so the flow was harder to break for UI-only changes. Both passed 50 of 50 runs, which makes stability a tie: the app is too simple to flake, and we make no fewer-flakes claim. This is one small app, so treat the numbers as one data point and not a benchmark.

The failure messages differed too. The flow’s named the step and the server’s answer, PostProducts … got 400: Product name, price and sku are required, and the UI test’s said element(s) not found.

The side-by-side page shows the journey as a short video and compares what each layer is for. Only the browser test would notice a broken Edit button, so keep it for what users see.

Find a flaky test, then contain it

Find flaky tests by running them more than once. Rerun failed tests on the same commit: a test that fails once and passes on the next run of the same commit is flaky by definition. Track which tests do this over a week and fix the worst first.

While you fix one, quarantine it. Move a known flaky test out of the required check so the rest of the suite stays trusted, and name an owner and a deadline, or the test stays out of the suite for good.

Repetition helps for API flows too. The ci command runs every flow several times and reports per step. In the Stresseur workflow, CI runs each flow 13 times, 1 warm-up and 12 measured. Repeated runs of the same flow are the raw material for spotting a step that passes in some runs and fails in others. Keep the whole thing fast enough to run on every pull request, as in running API tests on every pull request.

Conclusion

Name the cause before you pick the fix. Wait for conditions, not for time. Select by what users see. Give every run fresh data and every test its own state. Then move the checks that never needed a screen into API flows, and keep a small set of browser tests for the checks that did.