A fix that doesn't go through testing isn't finished
A fix written directly in production isn't a fix: it's a divergence waiting to snap back at you. It took just one character — an at sign turned into a dot — to break the header of twenty public pages.
In short: twenty pages of the showcase were repaired by hand, in production, without going through testing. The stylesheet address was copied with a typo:bootstrap.3.3instead ofbootstrap@5.3.3. The address returns 404, the style never loads, and the site menu falls back to a bulleted list. No one saw it, because the English version of the same pages was correct.
1. The pipeline that already existed
The tower publishes its showcase in two stages. You write in a test folder, a non-indexed address serves it, you look. Only then, and only after approval, does a copy go to production. This discipline dates from the day we realized the repository folder was production: a simple git pull published, and nothing separated "I write" from "the world sees it".
The fix in question didn't go through that. It was applied where pages are served, not where they are written.
2. Why the mistake slipped through
Fixing in production means fixing without a safety net. Three protections disappear at once.
Review disappears. In testing, the page is looked at before it exists for the public. In production, the first person to review is a visitor.
Comparison disappears. The publishing pipeline compares the fingerprint of what production serves to what testing validated. A change that only exists in production is, to it, indistinguishable from corruption — because that's what it is.
The symptom hides. Only the French pages received the faulty address; the English ones kept the correct one. Yet many monitoring tools, and many browsers configured in English, land on the English version. The site seemed healthy to whoever checked it, and broken to whoever read it.
3. The next trap, more serious than the first
Production is a copy of testing. A repair done by hand in production doesn't survive the next publication: the copy overwrites the original, without an error message, without a trace, often days later when no one makes the connection anymore.
Here, the story turns around: because testing had remained correct, it's the normal publication that repairs the twenty pages. But the opposite was entirely possible — a useful fix applied outside the pipeline, erased at the next publication, and a bug that comes back on its own without anyone understanding why.
4. What we do about it
- We don't rewrite production by hand. Never, even for one line, even when it's urgent. The testing → approval → production pipeline exists so that "it works on my machine" is never the last check.
- We check a resource's address, not just its presence. A well-formed tag pointing to a dead address looks exactly like a working tag. A check that reads the HTML will see nothing; a check that requests the resource will see the 404.
- We check both languages. A check that follows the browser's language only tests half the site — and gets the wrong half half the time.
- We detect divergence instead of waiting for it. Comparing the two folders costs a few seconds; discovering weeks later that they've diverged costs an evening.
The lesson fits in one sentence: a fix that doesn't go through the normal path isn't finished, it's displaced. It lives in the system's blind spot — and there it dies, or there it rots.