lecture 12 of 150 completed

The test that only fails on the build server

A step-by-step method for the failure you cannot reproduce on your own machine.

This will happen to you, probably in your first month. A test fails on the build server about once in ten runs. On your machine it passes every time. Someone re-runs the build, it goes green, and everyone moves on.

Do not move on. Here is the method.

Step 1. Make it fail for you.

You cannot fix something you cannot see. Run that one test many times:

npx cypress run --spec cypress/e2e/contacts.cy.js

Run it twenty times, or fifty. Most timing bugs show up within twenty runs.

Still passing? Make life harder for it. Open your browser dev tools and set the network to a slow speed, then run again. A slow network makes timing bugs appear quickly, and the build server is usually slower than your laptop.

Step 2. Look at what the build server saved.

A failed CI run is not a mystery. Cypress records a video of the run and takes a screenshot at the moment of failure. Open the screenshot first and ask one question:

What is on this screen that I did not expect?

Nine times out of ten the answer is right there. A loading spinner still turning. An empty table. A cookie banner sitting on top of the button you wanted to click.

Step 3. Find the exact line.

Open the run in Cypress and look at the command log on the left. Every command is listed with its result. Click a step and Cypress shows you the page as it was at that moment. Start at the red step and walk backwards until something looks wrong.

Step 4. Fix the cause, not the symptom.

There are three usual causes:

  • A fixed wait. cy.wait(400) was long enough on your machine and too short on the server. Replace it with a check or a named request.
  • Shared data. Another test left something behind. Give this test its own data.
  • A real bug in the app. The button can be clicked before the app is ready to handle it. Your user will hit this too. Report it.

Two things people do instead, and you should not. Raising the timeout hides the problem. Turning on retries hides the problem. In both cases the bug stays in the app, and now it is invisible.

Remember this: reproduce it, then narrow it, then fix the cause. A test you fixed without understanding will come back next month.

Reference: Debugging.

now prove it

Reading is the setup. Solve this problem in the editor. We break the app on purpose to check your test would catch it.