freedzhost.cloud

Does your test catch anything? Prove it by breaking the code (mutation testing)

Level AdvancedEstimated time 1 hRead first Giving code work to a local agent without it breaking everythingA word you don't know? Glossary

When an agent writes code and its tests, both can share the same blind spot: the test was written alongside the code and checks what the code does, not what it should do. A green result then proves nothing. Mutation testing asks a simpler question: if I break this line, does any test notice?

The codetests green firstOne mutant== becomes !=, -eq becomes -ne…Run the testsred: caughtgreen: survived → blind spotrestore, next mutant
Break the code on purpose, one small change at a time, and run the tests. Red means a test caught it: good. Green means the defect survived: the tests are blind there. Before starting, the tests must be green on the real code, or every mutation looks caught.

A bit of history

Mutation testing was proposed by Richard Lipton in the early 1970s and formalised in 1978 with DeMillo and Sayward: to judge a test suite, plant small deliberate bugs in the code and see whether the tests notice. A surviving mutant is a bug your tests would have let through. Long considered too costly, it is cheap today, and more useful than ever when code is written by models.

1. The method

  1. Run the tests on the real code: they must all be green. If one is already red, every mutation will look "caught" for an unrelated reason.
  2. Change one small thing: == into !=, -eq into -ne, < into <=, a condition into "always true", || exit into || true.
  3. Run the tests. Red: the defect is caught. Green: it survived, the tests are blind there.
  4. Restore the code exactly, and go to the next mutation.

Tools exist for most languages (mutmut for Python, for example). Our homelab is mostly shell scripts, so we wrote a small one that also mutates Bash operators; a mutation tool that only bites on Python measures nothing in a shell-script fleet.

2. What it found for us

A test suite of 68 checks, all green. Five mutations later, two blind spots: two failure paths were never exercised. Once filled, one mutation looked caught, but by a scenario that was already red before the mutation: a false detection. Final version: 81 checks, 11 defects injected, 11 caught.

Another time, a test checked that a word appeared in a file. Inverting one comparison in the code swapped two verdicts entirely, and the test stayed green. The fix was to make the test run the function that decides, not grep its output.

Traps we paid for

In Python, restoring a mutant with a file copy in the same second can leave the mutant's compiled .pyc in place: the next run tests the mutant again. Delete __pycache__ after each mutant, or run with PYTHONDONTWRITEBYTECODE=1.

In shell tests, OUT=$(my_function) runs the function in a subshell: an exit code it stores there is lost, and every refusal scenario passes green. Send the output through a file and keep the exit code in the current shell.

A surviving mutant is not always a defect: some changes cannot be detected by any test (an "equivalent mutant"). Read a low score as coverage, not as a grade, and look at each survivor before concluding.

Fixtures written from memory prove nothing about the real system: our tests were green on invented data while the code read fields that did not exist in the real API answer. Build fixtures from measured responses.

Sources

Found this useful? Tell us what to measure next.