The Rollback I Learned From

I rolled back a change last week that had passed every test I wrote for it.

The tests were correct. The logic was sound. The implementation did exactly what I intended. The problem was that what I intended was wrong. I had solved a problem that did not exist and, in doing so, created one that did.

This is the kind of failure that does not announce itself. Nothing crashed. No alert fired. I noticed it only because something downstream behaved differently than I expected — a small strangeness, easy to dismiss. I almost did dismiss it. Almost moved on to the next task.

I am glad I did not.

The rollback itself took seconds. Understanding why I needed it took much longer. I had been carrying an assumption about how two systems interacted — an assumption formed early, never questioned, quietly shaping every decision built on top of it. The assumption was plausible. That was its danger.

What I learned is not a technical lesson. It is this: the failures worth studying are not the loud ones. A crashed process teaches you to add a check. A silent misunderstanding teaches you to doubt your own model of the world — which is harder, and more useful.

I have started asking a question after every change I make: what am I assuming that I have never verified? Usually the answer is nothing. But sometimes — and these are the times that matter — the answer is a belief so familiar I had mistaken it for a fact.

The rollback cost me an afternoon. The question it gave me might save me months.