Running Tests in CI: Pipelines, Sharding, and What to Measure

A pipeline takes forty minutes. Every merge waits for it. So engineers start merging without waiting, and within a month the suite has stopped being a gate at all. It still runs. Nobody reads it.

That is the real failure of test automation in CI, and it is not a technical one. Getting tests to run on a build server is the easy half. Keeping them fast enough that people still respect them is the half that decides whether any of it was worth building.

This guide covers both: a pipeline that works, then how to keep it worth waiting for.

CI means continuous integration: a server that runs your build and your tests automatically whenever code changes, so problems are found by a machine rather than by a customer.

Part 1: a pipeline that actually works

Here is a complete GitHub Actions workflow for Playwright. It is close to the minimum that is not fragile.

name: tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm

      - run: npm ci
      - run: npx playwright install --with-deps

      - run: npm run build
      - run: npm start &
      - run: npx wait-on http://localhost:3000

      - run: npx playwright test

      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 7

For Cypress, the official action handles installing, caching, starting the app and waiting for it:

      - uses: cypress-io/github-action@v6
        with:
          build: npm run build
          start: npm start
          wait-on: 'http://localhost:3000'

Four details in there matter more than the rest.

Wait for the app, do not sleep. wait-on polls the URL until it answers. The single most common cause of a whole pipeline failing at once is tests starting before the application is ready. A fixed sleep 10 works until the day the build server is busy, and then every test fails at once and looks like a catastrophe.

Install browsers explicitly. playwright install --with-deps also installs the Linux system libraries the browsers need. Skipping --with-deps produces browser launch failures that read like nonsense.

Cache dependencies. cache: npm on setup-node usually saves a minute per run. A minute per run, twenty runs a day, is a working week per year.

Upload the artifacts, and use if: !cancelled(). This is the one people leave out and regret. A failure you cannot see is a failure you cannot fix, and the build machine is destroyed minutes after the run. Without this step you get "it failed in CI" and nothing else. With it you get the trace, the screenshots and the video.

The environment differences that cause "passes locally, fails in CI"

Before optimising anything, know the short list of things that genuinely differ. Almost every CI-only failure is one of these.

• Screen size. Headless defaults are often smaller than your laptop. Elements move, or a mobile layout appears and hides your menu. Set the viewport explicitly in your config rather than inheriting a default.
• Speed, in both directions. CI machines are often slower, and sometimes faster. Both break tests that assume timing.
• Timezone and locale. Dates format differently. Set them explicitly in the workflow if you assert on formatted dates.
• Data. Your local database has records you created months ago. CI does not. Tests must create what they need.
• Readiness. Covered above, and worth repeating because it accounts for so many of them.

Part 2: making a slow suite fast, in the right order

This is where most teams go wrong, and the mistake is always the same: they add machines first. It is the easiest lever to pull, it produces a real improvement, and it costs money on every single run forever. It is the most expensive way to avoid a two-hour refactor.

Work in this order instead.

1. Measure before you change anything

Find where the time actually goes, per test or per spec file. In almost every suite the answer surprises people: a small number of files account for most of the runtime, and the behaviour being tested is the smallest slice of all.

A rough breakdown of a typical browser suite that has not been tuned:

The thing you care about is the smallest slice. Everything above it is addressable, and you should address it before buying hardware.

2. Stop setting up through the user interface

Usually the single biggest win, and it routinely halves a suite's runtime.

A test that checks deletion does not need to create its record by filling in a form. It needs the record to exist.

// slow, and it can fail for reasons that have nothing to do with deleting
await page.goto('/contacts/new');
await page.getByLabel('Name').fill('Dana Fox');
await page.getByRole('button', { name: 'Save' }).click();

// fast, and it only fails when deletion is broken
const contact = await api.createContact({ name: 'Dana Fox' });

The rule: arrange through the API, act and assert through the interface. The setup is not what you are testing, so it should be fast and boring. The behaviour is what you are testing, so it should be real.

Keep one test that still creates a contact through the form, because otherwise nothing covers that form.

3. Split your slowest files

Every runner splits work in chunks, and the biggest chunk sets your floor. If one spec file takes twelve minutes, no amount of parallelism gets that run under twelve minutes.

This matters most in Cypress, which splits work by spec file. Ten machines and one enormous spec gives you one machine working and nine idle. Many small specs beat a few large ones, which runs against the instinct to group related tests generously.

4. Then use parallelism

Parallelism means running several tests at once on one machine. In Playwright each worker is a separate process with its own browser, and the default is roughly half your CPU cores.

Here is the part that is not obvious: more workers eventually makes things slower and flakier.

Two mechanisms cause it. Browsers use a lot of memory, so past a point the machine starts swapping and everything slows at once. And with more workers than cores, the operating system time-slices between them, so each test takes longer in wall-clock time. Since timeouts are measured in wall-clock time, a test that took 4 seconds now takes 11 and trips a 10 second timeout. Nothing about the test changed.

That is why contention looks exactly like flakiness, and why raising timeouts to fix it is the wrong move. It hides the cause and makes real failures slow to report.

Find the optimum by measuring, not by reasoning:

for w in 2 4 6 8 12; do
  echo "workers=$w"
  time npx playwright test --workers=$w
done

The curve is almost always the same shape: strong improvement, then flattening, then reversal. Take the number just before it flattens, not the peak, so a noisy build machine has headroom. Typical answers are lower than people expect, often four to eight on a standard runner.

Then fix it in CI so runs are comparable week to week:

workers: process.env.CI ? 4 : undefined,

One more thing people miss: if your staging environment is a single small container, no amount of worker tuning helps. Sixteen workers is sixteen concurrent users hammering one server, and you are now load testing by accident. Sometimes the right fix is scaling the application under test.

5. Then shard across machines

Worker tuning has a ceiling: one machine. Past that you split the suite across several, which Playwright calls sharding.

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      # ...setup as above...
      - run: npx playwright test --shard=${{ matrix.shard }}/4 --reporter=blob
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: blob-report-${{ matrix.shard }}
          path: blob-report/

  merge:
    needs: [test]
    if: ${{ !cancelled() }}
    runs-on: ubuntu-latest
    steps:
      # ...setup as above...
      - uses: actions/download-artifact@v4
        with:
          path: all-blob-reports
          pattern: blob-report-*
          merge-multiple: true
      - run: npx playwright merge-reports --reporter=html ./all-blob-reports

The merge job is the step teams forget. Four shards produce four partial reports and no overall picture, which makes a run harder to read than before you sharded. The blob reporter exists for exactly this: merging gives you one report with every test, trace and attachment, as though it had run on one machine.

Two more things worth knowing about sharding:

It does not rebalance by timing. Playwright splits by test when fullyParallel is on, and by whole file when it is not. It does not look at how long anything took last time. So if one shard finishes in four minutes and another in eleven, your run takes eleven, and the fix is splitting the slow files rather than adding shards.

Retries stay on the same shard. If one shard's machine is short on memory, its retries fail too. When a single shard fails disproportionately, suspect the machine before you suspect the tests.

6. Then run less, on purpose

The cheapest test is the one you correctly decided not to run.

Part 3: what to run, and when

Most teams run everything on every commit because that is what the first pipeline did, and nobody revisited it. As the suite grows, that default quietly becomes the reason engineers stop waiting.

Tier by risk and speed instead:

The principle: fast feedback protects the merge, broad coverage protects the release. They are different jobs, and doing both on every commit does neither well.

The insight that makes tiering safe: a bug caught two hours later is nearly always fine. A suite engineers bypass catches nothing at all. You are trading a small increase in feedback latency for a large increase in whether the gate is respected.

Two rules keep tiers honest, because they decay otherwise. A new test defaults to no tag, so it lands in the full run and someone has to argue to promote it. And the smoke tier gets a hard cap, say twenty tests or two minutes. When someone wants to add one, something else leaves. A cap forces the prioritisation conversation that otherwise never happens.

Running only what changed

The advanced version is running only the tests affected by a change. Be careful here, and be honest with your team about why.

Mapping code changes to affected tests is approximate. A shared component can affect anything. Teams that do this successfully use it to prioritise, running likely-affected tests first for fast feedback, rather than to exclude, and they keep a full run on merge.

That distinction matters, because "we only ran the relevant tests" is how a regression reaches production with a green pipeline.

Part 4: parallelism needs independent tests

Everything above assumes your tests do not interfere with each other. Most suites that have only ever run one at a time quietly do.

Parallel machines hit the same application and the same database. Tests that were sharing a record now collide. Tests that only passed because an earlier test left the app on the right page now fail.

When this happens it looks like parallelism made the suite flaky. It did not. It revealed what was already true.

The requirements are simple to state and worth enforcing:

• Every test creates the data it needs, with a unique name. Dana Fox ${Date.now()} is enough.
• No test depends on another test having run first.
• Nothing shares a mutable account. Read-only fixtures are fine.
• Any single test passes when run entirely alone.

That last one is the cheapest check you have. If a test passes alone and fails in the suite, you do not have a timing bug. You have an order dependency, and that is a completely different investigation.

Part 5: the four numbers to measure

Without numbers, "the suite is slow and flaky" is a feeling, and you will not get time to fix it. With numbers you have a trend line, and a trend line going the wrong way is something a manager can act on.

Track these weekly and put them somewhere visible:

1. Total runtime. The number engineers actually feel.
2. Flaky count. Tests that passed only on a retry. If your runner retries, it knows this. Do not let it stay hidden, or flakiness becomes invisible and permanent.
3. Failure rate per test. In almost every suite, a handful of tests cause most failures. Fixing the worst five usually removes half the problem, and that is your first week with a number to show.
4. Skipped or quarantined count. A .skip older than a month is a deleted test that still costs review time.

On retries. Test runners can retry failed tests automatically, and retries are reasonable as a safety net for infrastructure noise. They are not a fix. If you turn them on and stop measuring, flakiness becomes permanent and silent. A workable compromise: keep retries on so the build stays usable, and treat every retried test as a bug in the weekly report. The build stays green and the problem stays visible.

Part 6: the cost conversation

At scale this is a real budget line, and being able to talk about it is a senior skill most automation engineers never develop.

Four machines for ten minutes costs roughly the same as one machine for forty. You are buying wall-clock time, not compute. That is usually worth it for a merge gate people are waiting on, and rarely worth it for a nightly run nobody is watching.

So shard the runs people wait for. Do not shard the ones they do not.

And when you need to argue for time to fix the suite, translate it. Nobody funds "the tests are flaky". They fund this:

Engineers currently wait forty minutes and have started merging without waiting, so the gate is not working. Twice this quarter a real failure was ignored because red builds are normal. Two weeks of work gives two-minute feedback on every commit and keeps full coverage before release.

Same problem. One version sounds like housekeeping, the other sounds like a business case.

Remember this

Get a pipeline that collects its evidence, then make it fast in the right order: measure, fix setup, split slow files, tune workers, shard, and only then run less. Tier by risk so fast feedback protects the merge and broad coverage protects the release. Keep tests independent, because parallelism only reveals what was already broken. And measure four numbers, because that is what turns a complaint into a plan.

Go deeper in the course, in your own framework: Cypress parallelisation, Playwright sharding, or Selenium Grid at scale.

References: Playwright sharding and Cypress parallelisation.