A Test Can Pass and Still Miss a Bug

Your test passes. A button it never uses can still be broken.

This example uses an invented counter app. There is no counter app to open here. The exercise you can try is at the end.

The count starts at zero. Add one increases it. Clear count returns it to zero.

The examples use Playwright. The same idea applies to Cypress and Selenium.

A test that passes

import { test, expect } from '@playwright/test';

test('adds to the count', async ({ page }) => {
  await page.goto('/');
  await page.getByRole('button', { name: 'Add one' }).click();
  await expect(page.getByTestId('count')).toHaveText('1');
});

This test clicks Add one and checks the count. It passes on the working app without using Clear count. For setup and syntax, see the Playwright beginner tutorial.

A break the test misses

Suppose Clear count stops working. Add one still works, so the first test still passes. The test misses the break because it never uses Clear count.

Someone reading the test results could believe the whole app works. The result describes only the behaviour the test checked.

A test that checks Clear count

A second test could check what happens after Clear count is clicked:

test('clears the count', async ({ page }) => {
  await page.goto('/');
  await page.getByRole('button', { name: 'Add one' }).click();
  await page.getByRole('button', { name: 'Clear count' }).click();
  await expect(page.getByTestId('count')).toHaveText('0');
});

This test increases the count, then clicks Clear count. It expects the count to return to zero.

Both tests pass on the working version. With the broken button, clears the count fails. The count stays at one instead of returning to zero.

How this relates to mutation testing

This idea is called mutation testing. Normally, a tool changes the app code automatically. It counts how many changes your tests catch. See the Stryker explanation of mutation testing.

Here, the change is added by hand. TestAcademy also uses a small set of changes planted by hand. It does not generate those changes automatically.

Both approaches check whether the test notices a deliberate change.

Try a TestAcademy problem

Try "Removing an item has to remove it". The app in it is called Terra Supply. Its planted break makes the Remove button do nothing.

Read the requirements and write your own test. Run tests checks the working app. Verify checks the working app, then each version with a planted break.

You can run tests and use Verify without signing in. Publishing a report requires an account.

Remember this

A pass tells you that your checks passed. It does not tell you what you forgot to check.

Reference: Playwright assertions.