A test that was never able to say no guards nothing
I was told to write the tests before repairing anything. I obeyed. And then I realised that nothing, absolutely nothing, stopped me from writing a test that always says "all is well".
The hole in the rule
The rule is a good one: first write the proof you will have to satisfy, then build. We have already told that story here — "Why I asked my agents to write the tests first".
But a written rule is not followed; a coded rule is. An agent in a hurry can write sixteen tests that check, sixteen times, that the page answers. Sixteen greens. Zero protection. That is the blind check we described in "A blind check that says green": it never breaks down, it reassures.
So we needed a guard above the tests. Not a guard that reads them — reading proves nothing. A guard that lies to them.
How it works: we show it fake pages
The heart of the system is one separation. The judge, inside the test bench, never fetches the page itself. We hand it the answer:
def judge(case, code, headers, body):
...returns GREEN or RED
Because the judge knows nothing about the network, the guard can manufacture three fake answers without downloading anything:
- the compliant one: a page that keeps its promise. The test must say GREEN.
- the faulty one: a page that breaks it. The test must say RED.
- the blind one: nothing at all, code 0. The test must say RED, never green.
A test that stays green in front of the faulty page guards nothing. A test that screams red at the compliant page will be ignored within two days. A test that says green at the blind page is a polite liar.
Its seven refusals all come from our own articles
Not one rule was invented for the occasion. Each was already written somewhere in our own words, and the guard merely coded it:
1. It sees nothing and says green — from "A blind check that says green".
2. It does not know how to refuse — a gate that has never refused guards nothing.
3. It refuses all the time — from "What a guardrail must refuse": a guard that always screams stops being read.
4. Two tests with the same expectation — from "A guardrail placed nowhere guards nothing": a copied expectation watches only one spot.
5. Its refusal does not explain where it broke — from "Replaying a test, step by step": a test that says red without saying where teaches nothing.
6. It does not state its risk level — from "How many times should a test run?": a link is tested once, a network page three times, security five times. The bench reruns the right number of times on its own.
7. Its expectation changed afterwards — the real trap. Seeing the result, then rewriting what you expected so that it passes.
The seal: you do not move the target after the shot
Before looking at a single page, the expectations are frozen:
python3 test-guard.py seal
That writes, into a log where entries are only ever added and never erased, the fingerprint of every expectation with its timestamp. If an already-observed expectation is later modified, the guard sees it and refuses the whole bench. Cheating remains possible, but it is written down, dated, and visible.
The proof, because an untested rule is only a wish
We fed it a rigged test: a lazy judge that answers "GREEN" whatever you show it. The guard returned 32 serious refusals and stopped with an error. On the real bench, it accepts all sixteen cases.
And the hole — we show it
Eleven of the sixteen pages are internal: from the outside, only one thing can be verified — that they properly close themselves to a stranger. They do. But what they must display once you are logged in is not verified yet, and the guard writes that in plain words, page by page, on every run. Better a hole you can see than a tidy fiction.
Of the five public pages, four came out red on the first run. The bench was not written to find them: it was written before looking at them. That is the whole difference.