lecture 13 of 160 completed

Diagnosing a failure you cannot reproduce

It fails in CI and passes a hundred times locally. A method beats guessing, and this is the method.

This is the defining senior skill, so it deserves a method rather than advice.

A test fails on the build server. You run it locally a hundred times and it passes every time. You have no reproduction, and the machine it failed on no longer exists. Everything you get is what was recorded.

Start by classifying the failure. The class determines the investigation.

Class one: it fails alone, consistently, in CI only. Something about the environment differs. This is the most tractable class.

Class two: it passes alone, fails in the suite. Another test is affecting it. Nothing about the environment is relevant.

Class three: it fails intermittently everywhere, including locally. A genuine race in the test or the app.

Class four: it started failing at a specific point in time. Something changed. Find the change.

Getting the class right saves hours, because each one has a completely different next step. Most wasted debugging time is spent investigating class two as though it were class one.

Class one: environment. The list of things that differ between your machine and CI is short and worth knowing by heart. Screen size (headless defaults are often smaller, which moves elements or switches to a mobile layout). Speed, both faster and slower. Timezone and locale. Available test data. Network latency to the app. Whether the app was fully started when tests began.

The last one is the most common cause of a wholesale failure and the easiest to fix: wait for a health endpoint before running, rather than assuming a fixed delay is enough.

Class two: cross-test interference. Do not read the failing test first. Find the test that broke it:

npx cypress run --spec "cypress/e2e/contacts/**"     # does the group fail?

Then bisect the group. Half the specs, run, repeat. Four rounds narrows fifty specs to one. This is mechanical and much faster than reading code, and it ends with a specific culprit rather than a theory.

The cause is almost always one of three things: shared data mutated by an earlier test, a cached session an earlier test changed, or state left in localStorage.

Class three: a real race. Reproduce it by making conditions worse rather than by running it more.

cy.intercept('GET', '/api/contacts', (req) => req.reply({ delay: 2000 }));

Slowing the response deliberately will often turn a one-in-fifty failure into a reliable one. A race that only appears under load is still a race; you are just making the window bigger so you can see it.

Class four: a change. Find the last passing run and the first failing one, then read what landed between them. This is usually the fastest of all four and people skip it because it feels like it should be harder.

What to collect so this is possible at all. You cannot investigate evidence you did not gather. At minimum: screenshots on failure, video for the failing spec, the browser console log, and the request log. Cypress Cloud collects these centrally, and if you are not using it, upload them as CI artifacts. Evidence written to a machine that gets destroyed is evidence you did not collect.

Reading the Cypress command log properly. The habit that speeds this up most: do not start at the red step. Start three steps earlier and ask what the page looked like then. The failure is usually a consequence, and the cause is upstream. Time-travel debugging in the runner exists for exactly this.

One rule about retries. If a test passes on retry, that information must not be discarded. Cypress reports it as flaky. Track that number. A test passing on retry is a test that will fail for a real reason one day and nobody will notice.

Remember this: classify first, bisect for interference, exaggerate conditions for races, and never investigate evidence you forgot to collect.

Reference: Debugging.

now prove it

Reading is the setup. Solve these problems in the editor. We break the app on purpose to check your test would catch it.