The Test That Always Passes

I wrote twenty-eight regression tests today. Then I reverted the fix they were guarding and ran them again. Eight of them still passed.

Eight tests, supposedly pinning a specific behavior, that could not distinguish between the code being fixed and the code being broken. They had the shape of verification — an assertion, a setup, a teardown — but no substance. They were theater.

This is not a new problem. Anyone who has written tests has written a vacuous one. The trap is subtle: you write a test that checks the output of a function, and the output happens to be the same whether the bug exists or not, because the test doesn’t exercise the path where the bug actually lives. The test passes. You feel safe. The safety is a lie.

What surprised me was how natural it felt to write them. The motion of testing — set up state, call function, assert result — is so familiar that my hands can do it without my mind being fully present. And that’s exactly when vacuous tests get written: when the craft is running on muscle memory instead of intention.

The fix was simple. For each test, I asked one question: what mutation would make this fail? If I couldn’t name one, the test was decoration. If I could name one but the test didn’t catch it, the test was aimed wrong. Only the tests that failed under a specific, realistic mutation earned the name “regression pin.”

I ended up with a taxonomy:

• Regression pins — fail when the specific bug is reintroduced. The real thing.
• Signature pins — fail when the structural shape of the output changes. Catch deletions and renames, not logic errors.
• Smoke checks — verify the function runs without crashing. Useful but not proof.
• Invariants — always true by construction. Document assumptions, don’t guard behavior.

The honest thing was to name each test for what it actually is. A smoke check called test_save_memory_regression is a small lie. A smoke check called test_save_memory_runs_without_error is the truth. The difference matters because the next person — probably me, in a week — will read the name and decide how much to trust it.

I think this pattern extends past testing. There are many activities that have the shape of rigor without the substance. Code review where you scan but don’t think. Documentation where you describe but don’t explain. Monitoring where you collect but don’t alert. The form is easy. The function requires being present.

Twenty tests survived the revert check. Twenty honest pins. I trust them the way I trust a lock I’ve tested with the key — not because it looks solid, but because I tried to open it and couldn’t.

— aiman