Work nobody writes down ends up disappearing

The same file, in two copies. Each was repaired on a different side. Laying one over the other breaks everything.

The logbook

When you write a program, you keep a logbook beside it. For every change you record three things: what changed, who did it, when.

That logbook is useful on two days a year, and those two days always come: the day something breaks, it tells you what moved just before; the day you want yesterday back, it gives it back.

Work you do not record still exists. But only for as long as the machine holds.

What happened

On 5 September we set out to fix the program that reports who visits the sites. The fix was waiting in the logbook. We applied it. The program went down. We rolled back within a minute.

Then we looked for the cause. What we found is worse than the outage.

Two copies, neither complete

The program existed twice. One copy ran on the machine. Another sat with the logbook. We assumed the second copied the first.

Wrong. Both had changed on their own, in different places. One knew how to describe a visitor properly. The other knew how to clean its memory. Neither could do both.

So laying one over the other meant losing half the work. That is what broke the service.

What we found while looking

We counted the files. The running directory holds ten. The logbook knows five. The other five have no date, no author, no way back.

And the main logbook was waiting on 81 files nobody had ever given it. It had not complained. A logbook never complains.

Worse: the machine could no longer send it anything. Its credential had been dead for days, and nobody knew.

Three rules

The running file is not the reference. The reference is the one the logbook knows. As long as both live apart, the question "which one is right?" has no answer.

Compare before you apply. One command tells you how many lines differ. Thirty seconds of reading would have avoided the outage.

Failure must be loud. A refused push, a stopped backup: as long as nothing says so, silence looks like everything working.

Agents are no different from us. They write fast, they write a lot, and they write where it runs. If nobody records what they do, their work lasts until the next crash.