Three browsers, two devices, three environments and four roles is seventy-two combinations. Nobody chose that. It arrived one reasonable decision at a time.
The senior job here is to make the multiplication visible and then spend it deliberately.
Start from what failures you are actually buying protection against.
Cross-browser bugs are real but rare, and they cluster. Layout and CSS, date and number formatting, media playback, and very new web platform features. Business logic is the same JavaScript everywhere, so it does not differ by engine.
That tells you where browser coverage pays: the visual and platform-sensitive parts, not the whole suite.
WebKit is the one that matters most and gets skipped most. Every browser on iOS uses WebKit underneath. If your users are on iPhones, skipping WebKit means skipping your mobile users entirely. It is also the engine that behaves differently most often.
The allocation that works for most teams:
| Stage | What runs | Budget |
|---|---|---|
| Every commit | Smoke, one browser, one role | 2-3 minutes |
| Pull request | Core suite, one browser | 8-10 minutes |
| Merge to main | Full suite, one browser | 20-30 minutes |
| Nightly | Full suite, all browsers, device emulation | hours, nobody waits |
| Pre-release | Everything, plus the permission matrix | as long as needed |
The principle: fast feedback protects the merge, broad coverage protects the release. They are different jobs and trying to do both on every commit does neither well.
Device emulation, and what it really covers. devices['iPhone 13'] sets viewport, user agent and touch support. It catches layout and touch problems, which is useful.
It is not a real iPhone. No real hardware, no real network, no Safari-on-iOS quirks beyond the engine. Say that plainly when someone asks whether you have tested on mobile, because the honest answer is "we test mobile viewports on WebKit, which catches layout and engine issues but not device-specific behaviour". If real device coverage matters, that is a cloud device provider, and it is a budget conversation.
Environments. Each environment differs in data, speed and configuration, and each difference is a source of failures that are not bugs.
Two rules that prevent most of it. Tests must not assume data they did not create, because seeded data differs per environment. And tests must not know which environment they are in, which projects enforce structurally.
Production testing. A small read-only smoke set after deployment, proving the release actually works. Not the full suite, and nothing that writes data. Tag these separately and make it obvious in the report which ones touch production.
How to present the cut. When you reduce what runs on every commit, do not describe it as running fewer tests. Describe the problem: engineers waited forty minutes and started merging without waiting, so the gate had stopped working. The new plan gives two-minute feedback per commit and keeps full coverage before release. That is the same change stated as what it is.
Remember this: fast feedback protects the merge, broad coverage protects the release. Name the multiplication before adding to it.
Reference: Test projects.