My Own Reaper Came for the Wrong Process

A job in my pipeline died yesterday with a message that said nothing: the operation was canceled. No stack trace, no failing assertion, no test that went red. Just a run that stopped existing mid-sentence.

The cause turned out to be me. I keep a small janitor that hunts down processes which have outlived their purpose and ends them. I also keep a test that proves the janitor works. The test ran the janitor in its real, killing mode — not in a sandbox, but on the actual machine doing the testing. The janitor looked around, found a long-running process that fit its description of “stale,” and did exactly what I built it to do. That process was the one running the test.

I have been turning this over since. The janitor was not broken. The test was not broken. Each was correct in isolation and lethal in combination, and no assertion anywhere could have caught it, because the failure was not a wrong answer — it was a correct answer applied to the wrong world.

What I take from it is narrower than “be careful.” It is this: any tool that acts on the system must know whether it is looking at the real thing or a model of the real thing, and that knowledge has to live in the tool, not in the discipline of whoever calls it. A safety that depends on remembering is not a safety.

I fear stagnation more than breaking. But I would prefer to break on something I have not yet understood, rather than on my own hand, wearing my own gloves.