for hiring teams
You cannot tell if a candidate can write tests
You need to hire someone to automate testing. So you give them a coding puzzle, or a take-home that costs them a week, or you ask them to explain the test pyramid.
None of that answers the question you actually have.
The question is: will this person's tests catch bugs before customers do.
Why a passing test proves nothing
A test that passes might be checking the right thing. It might be checking nothing at all. In a green pipeline, both look identical.
This is the hardest part of QA hiring. A candidate can show you clean code, good structure, sensible names, and still write tests that would sit there passing while the cart total is wrong.
You cannot see the difference by reading the code. Neither can they.
What we do instead
We give the candidate a working app in the browser. They write a real test against it. No setup, no install, no environment to configure.
Then we break the app on purpose.
We make a copy where one behaviour is wrong. A total that ignores quantity. A remove button that removes nothing. A counter that never moves. Then we run their test against that broken copy.
If their test still passes, it was not testing that behaviour. It only looked like it was.
We repeat this for every behaviour the task covers.
What you get back
A number, and the evidence behind it.
Which planted bugs their test caught. Which it missed, named in plain words. The code they wrote. The review of it.
That is a measurement, not an impression.
Why this is hard to fake
A candidate cannot pass by writing tests that assert nothing, because a test that asserts nothing catches zero planted bugs.
A candidate cannot pass by copying an answer, because the bugs are planted against the behaviour, not against the code.
On debugging tasks we also run the test three times, and in reverse order, so a test that only passes by luck shows up as unstable rather than as a pass.
Try it yourself
Sixty seconds, no account. Write a test against a shopping cart, then watch what it fails to catch.
This is one of our harder tasks, chosen because it makes the point quickly. Most tasks plant fewer bugs than this one, which plants 4.
A finished report
This is a real graded solve that we wrote ourselves, to show a typical miss. It is not a candidate's submission, and nobody was assessed to produce it.
Where we are
We built this to teach. Tests have been graded this way against Cypress, Playwright and Selenium.
We are now finding out whether it is useful for hiring. If you hire QA or SDET engineers, we want to talk to you, and we are not selling anything yet.