You already know how to fix a single flaky test. This lecture is about the situation you will inherit: a suite where forty tests fail at random and everyone has given up.
You cannot debug forty tests one by one. You need a different approach.
Step 1: measure before you fix anything. Playwright's HTML report and JSON reporter both record which tests were retried. Collect that over a week and count failures per test.
Do not skip this. Without numbers, "the suite is flaky" is a feeling, and you cannot show progress to a manager who is asking why testing takes so long.
Step 2: rank by pain, not by ease. Sort tests by how often they fail. In almost every suite I have measured, a small number of tests cause most of the failures. Fixing the worst five usually removes half the problem. That is your first week, and it gives you a number to show.
Step 3: look for the shared cause. Flaky tests come in families. When you fix five, you will notice the same causes underneath:
- Values pulled into variables with
textContent()instead of checked with a retryingexpect. page.waitForTimeoutcopied from an old test into fifty new ones.- Shared data or a shared account that parallel workers fight over.
That last one deserves attention in Playwright, because it runs files in parallel by default. A suite that was fine on one worker can become flaky the day someone increases the worker count, and the tests did not change at all.
Step 4: be careful with retries. Playwright makes retries easy, and that is a trap. Retries are useful for keeping the build usable while you fix things. They are not a fix. If you turn on retries and stop measuring, the flakiness becomes invisible and permanent.
A good compromise: keep retries on, but treat every retried test as a bug in a weekly report. The build stays green, and the problem stays visible.
Step 5: agree a rule for new flakiness. A rule that works: a test that is retried twice in a week gets fixed or removed that week, by the person who owns that area.
Here is the hardest part, and it is not technical. You will need time from your manager, and "fixing flaky tests" sounds like tidying up. Frame it in their language: how many developer hours per week are lost re-running builds, how long a release is delayed, what a missed bug costs.
Remember this: measure first, fix the shared causes, and never let retries hide the numbers.
Reference: Test retries.