Your build goes red. You look at the failure, shrug, and press "re-run". It passes. Nobody investigates, because everybody already knows that test "is just flaky".
That habit is how a real bug reaches production. Once a team learns to ignore a red suite, the suite has stopped doing its job.
The good news is that flakiness is not bad luck. It has a short list of real causes, and each one has a known fix.
What is a flaky test?
A flaky test is a test that passes sometimes and fails other times, with no change to the code and no change to the test. Run it ten times and you might get eight passes and two failures.
The important word is nondeterministic. A normal failing test tells you something is broken. A flaky test tells you nothing at all, because the result depends on something the test does not control: how fast the machine was, what order the tests ran in, or what another test left behind.
That is what makes flaky tests worse than plain failures. A failing test is information. A flaky test is noise that looks like information.
Why flaky tests cost more than they look like they do
The obvious cost is time. Somebody re-runs the build, waits, and gets on with their day.
The real costs are quieter:
- The suite stops being a signal. When a third of failures are noise, people stop reading failures. Real bugs then travel with the flaky ones.
- Releases slow down. Every red build needs a human to decide whether it matters.
- Coverage quietly shrinks. Flaky tests get skipped, and a skipped test protects nothing.
I have watched a team spend a full afternoon chasing a failure in a checkout test, convinced they had a race condition in the payment code. The test was asserting on a total that a different test had changed. The application was fine the whole time.
Cause 1: Timing. The test is faster than the app
This is the number one cause by a wide margin. The app needs 800ms to load data. The test checks the result after 200ms. It fails. On a faster day, it passes.
The wrong fix is a sleep:
await page.waitForTimeout(3000); // do not do this
A sleep is a bet on a number. On a slow day 3 seconds is not enough. On a fast day you wasted 3 seconds. A suite full of sleeps is both slow and flaky.
There is research worth knowing here. Developers often believe they fixed a flaky test by increasing a timeout, when measurement shows the change had no real effect. The test got slower, not more reliable.
The right fix is waiting for a condition:
// Playwright: assertions retry automatically
await expect(page.getByTestId('toast')).toBeVisible();
// Cypress: should() retries automatically
cy.get('[data-testid="toast"]').should('be.visible');
// Selenium: use an explicit wait
await driver.wait(until.elementLocated(By.css('[data-testid="toast"]')), 5000);
The rule: wait for what you expect, never for an amount of time.
Cause 2: Bad selectors
A selector like this breaks whenever the layout changes:
page.locator('div > div:nth-child(3) > span.text-sm');
And a selector that matches several elements may hit a different one depending on rendering order. That is the sneaky version, because it fails intermittently rather than always.
Fix:
- Prefer stable, user-facing locators: roles, labels,
data-testid. - Make selectors match exactly one element. If a locator can match many, narrow it with a filter, and understand why there are many.
page.getByTestId('product-card')
.filter({ hasText: 'Alpine Tent' })
.getByRole('button', { name: 'Add to cart' });
Cause 3: Shared state between tests
Test A creates a contact. Test B counts contacts and expects 8. Run them together and B sees 9. Run B alone and it sees 8. You get random failures that depend on order and on parallelism.
Fix:
- Every test prepares its own data, or the suite resets state before each test.
- Never assert on a global count that another test can change. Assert on the thing this test created.
- Do not share logins, carts or records between tests unless they are read-only.
Cause 4: Test order dependence
This is the close cousin of shared state. A test passes only because an earlier test left the app on the right page or created the right record. The day someone runs the suite in parallel, or filters down to one test, everything breaks.
Fix: each test starts from a known point, usually a fresh page load and its own setup in a beforeEach hook.
The quick check is simple. Run any single test on its own. It must pass. If it only passes as part of the full suite, it is order dependent.
Cause 5: Environment differences
The test passes locally and fails in CI. The usual reasons:
- Different screen size. An element sits behind a hamburger menu in CI's smaller window.
- Slower machines. CI runners are often slower and busier than your laptop, so timing problems appear there first.
- Different data. The staging database changed under you.
- Animations. An element is still sliding into place when the test clicks it.
Fix:
- Set an explicit viewport size in the test config, so every run agrees on the window.
- Fix timing with condition-based waits. They absorb slow machines automatically.
- Make tests create the data they need instead of expecting it to exist.
- Disable or shorten animations in the test environment.
Cause 6: Parallelism and concurrency
Running tests in parallel is how you keep a suite fast. It also exposes every hidden assumption at once.
Two tests that both edit the same user, or both write to the same file, will pass alone and fail together. Worse, they fail in whichever order the runner happened to pick, so the failure moves around.
Fix: give each parallel worker its own data. A common pattern is to key test data by the worker index or a random suffix, so two workers can never collide.
Cause 7: Real time, time zones and randomness
Any test that depends on the actual clock will eventually fail on its own.
- A test that creates a record "today" and asserts on a date breaks when the run crosses midnight.
- A test written in one time zone fails when CI runs in UTC.
- A test that asserts on the first item of a randomly ordered list fails whenever the order changes.
Fix: control the inputs. Freeze the clock where your framework supports it, pass explicit dates rather than relying on "now", and sort data before asserting on position.
Cause 8: Services you do not control
If your test calls a real payment provider, a maps API or a partner feed, then their downtime is your red build. Their rate limit is your flaky test.
Fix: mock what you do not own. Intercept the request and return a fixed response, so the test measures your code rather than someone else's uptime.
// Playwright: return a fixed response, so the test never depends on the network
await page.route('**/api/products', (route) => route.fulfill({ json: [] }));
This is also the only practical way to test error states. Making a real server return a 500 on demand is hard. Faking one takes a line.
How to find flaky tests before your team stops trusting the suite
Most teams discover flakiness by noticing that people have started ignoring failures. That is far too late. Find them on purpose instead.
Re-run the suite on an unchanged commit. Nothing changed, so anything that fails is flaky by definition. Run this nightly and you have a flakiness report with no extra tooling.
Run the suite in a random order. If results change with order, you have order dependence.
Watch for tests that fail and then pass on retry. Most runners record this. A test that needed a retry is a flaky test, even though the build went green.
Track it per test, over time. One failure is noise. The same test failing in 4% of runs for a month is a bug with an address.
How to debug one flaky test
- Reproduce it. Run the single test 20 to 50 times in a loop. If it fails three times, you can work on it. A failure you cannot reproduce is one you cannot verify you fixed.
- Read the failure precisely. Which step, which locator, what was actually on the page? Screenshots, videos and traces answer this. A Playwright trace shows the DOM before and after every action.
- Look for the race. Ask what the test assumed was already true at that moment. That assumption is almost always the bug.
- Fix the cause, not the symptom. A longer timeout or an extra retry hides the problem, and it comes back later in a worse place.
Should you use automatic retries?
Every major runner can retry a failed test automatically. This splits opinion, so here is the honest version.
Retries are reasonable as a safety net for genuine infrastructure noise: a dropped connection, a runner that lost its network for a second.
Retries are harmful when they become the fix. If a test only passes because it ran three times, the suite is now lying to you, and the underlying race is still in your application or your test.
Two rules keep retries honest:
- A retried pass must be visible. If the report shows a green build and hides the retry, nobody will ever fix anything.
- Every retried test is a ticket. Treat it as a bug with a known reproduction, not as a solved problem.
Quarantine: what to do with a flaky test you cannot fix today
Sometimes you find a flaky test on the afternoon of a release. You cannot fix it now, and you should not delete it, because it covers something real.
Quarantine it. Move it out of the blocking pipeline into a separate job that still runs and still reports, but does not stop a release.
Quarantine works only with a rule attached: a date, and an owner. A quarantine folder with no expiry becomes a graveyard, and the coverage you thought you had is gone. Give each quarantined test a deadline, and delete it if the deadline passes without a fix. A smaller suite you trust beats a bigger one you ignore.
How to measure flakiness
You cannot manage what you do not count. Two numbers are enough to start.
Flake rate per test. Out of the last 100 runs on unchanged code, how many failed? Anything above zero deserves a look. Anything above 2% deserves a ticket.
Share of red builds that were noise. Out of every failed build last month, how many turned out to be a real bug? If that number drops under half, your team is already learning to ignore failures.
Best practices summary
- Wait for conditions, never for an amount of time.
- One test, one behaviour, its own data.
- Selectors should be stable, user-facing, and match exactly one element.
- Any single test must pass when run alone.
- Mock the services you do not control.
- Fix flaky tests quickly, or quarantine them with a deadline. An ignored red suite is worse than a smaller green one.
Frequently asked questions
What is an acceptable level of flakiness? Zero is the only target that keeps a team reading failures. In practice, teams track a flake rate per test and treat anything above about 2% of runs as a bug to fix.
Are retries a fix for flaky tests? No. Retries are a safety net for infrastructure noise. If a test needs a retry to pass regularly, the cause is still there and the report is hiding it.
Why does my test pass locally but fail in CI? Usually a different screen size, a slower machine, or data that exists on your laptop and not on the runner. Set an explicit viewport, use condition-based waits, and let each test create its own data.
How do I know if a test is flaky or the app is broken? Re-run the suite on a commit where nothing changed. Real bugs fail every time. Flaky tests fail some of the time.
Should I delete a flaky test? Only after trying to fix it, and only if you accept losing the coverage. Quarantine first, with an owner and a deadline, so the decision is deliberate rather than accidental.
Remember this
Flaky tests are not random. They come from a short list of causes, and timing is the biggest one by far. Wait for conditions rather than for seconds, give every test its own data, and never let a retry stand in for a fix.
Practice these patterns in the practice editor. The template apps have real loading delays, so a lazy assertion genuinely fails and a condition-based wait genuinely fixes it.
Related reading: waiting in Cypress, Playwright and Selenium is the full reference for the waiting rules above. Two flaky failures are common enough to have their own guides: StaleElementReferenceException in Selenium and element detached from the DOM in Cypress. If tests only turn red on the build server, running tests in CI covers what changes there.