lecture 17 of 180 completed

When your tests run without you

What changes on a build server, and why a slow or flaky suite quietly becomes useless.

You write tests on your laptop. They do their real job on a build server: a machine that runs your whole suite automatically whenever someone changes the code. Most teams call this CI.

Four things are different there, and they surprise almost everyone.

The machine is slower, and there is no screen. Build servers are usually smaller than your laptop and run the browser without showing it. Tests that quietly depended on your fast machine start failing. That is not bad luck. The server is finding timing assumptions you did not know you had made.

Tests run at the same time. Playwright runs files in parallel by default, and most teams also split the suite across machines. Every rule about independent tests and unique data matters here. A suite that only works in one fixed order breaks the day someone adds a machine.

Nobody is watching. All you get is what the run saved. Keep traces on for retries, and make sure your team knows where to download them.

A slow suite gets ignored. This one is about people, not code. If the suite takes forty minutes, developers stop waiting for it. They merge and check later. Then they stop checking. Now you have a suite that costs money and catches nothing.

Keep it fast: save the login with storageState, create data through the API, mock the slow states, and keep browser tests for flows that truly need a browser.

And the worst problem: a flaky test. A test that fails at random teaches your team a bad habit. They see red, re-run it, it goes green. After a month they re-run every red build without looking.

I have seen this story finish badly. A payment bug reached production on a Friday afternoon. The build had caught it. Three people had re-run that build without opening the report, because everyone had learned that red usually meant nothing.

Treat a flaky test as a bug. Fix it or delete it. A test nobody believes is worse than no test.

Remember this: your suite is useful only when the team trusts it and waits for it. Speed and stability are the job, not extras.

Reference: Continuous Integration.