An audit that got it wrong three times out of four — and why that is good news
Alice is an intelligence that lives at my place, on a machine of mine. She reads documents, cuts them into passages, and finds the right passage when you ask her a question. One morning an agent audited her and reported four defects.
Only one was real.
The other three came from assumptions nobody had checked. This article tells the two real ones, and above all the three false ones — because a false report costs more than no report at all.
The real defect: she found the answer, then threw it away
Here is her log, line by line, for the question "how does the router work?". A "log", here, is the trail the program writes while it works.
MAP : 'Router' -> confidence 1.00 MAP : selects Router (score: 1.00) MAP : found but with no steps -> moving on MAP : not found MODEL: calling Qwen...
Read the third line, then the fourth. She finds it. Then she declares she found nothing, and wakes up her large language model.
The cause fits in one rule in the code: "if this point has no steps, act as if you had seen nothing." But out of the 77 points her map can return:
with "steps" 34 no steps, but with a "detail" 43 <- all thrown away neither one nor the other 0
The "detail" is precisely the answer. She was throwing away more than half of what she knew, and paying for a call to her large model to retrieve something worse.
What changed: when the point found has no steps but carries a detail, we return that detail.
The proof, measured before and after:
before: source=new decision=model 3.99 s
"It connects your devices to your WiFi."
after : source=map decision=circuit 0.22 s
"Router: routeur.py:126 conductor THRESHOLD 0.3"
Eighteen times faster, and it is her answer, not a model's guess. The test written for it passes 7 green, 0 red. It also checks that a question outside her map still goes to the model: a map that answered everything would be worse than the original defect.
The second real defect: 853 lines were running unsaved
Her machine had two folders. They are not two rival versions: one is the repository that keeps the history of the code, the other is the code that actually runs.
Comparing the seven files: four differed, one of which had never been added to the repository at all. 853 lines were living without a net. A disk failure would have erased several days of work — including two abilities she had just been taught.
That is repaired: the four files went into the repository and were recorded there. Sending them outside has not been done — sending is publishing, and publishing is decided, not slipped in.
What holds: she refuses to invent
Faced with a question she has no answer to, Alice replies "I don't know." Verified again after the repair. The backup of her code carries the name of the day she was taught that. The guardrail holds, and the repair did not damage it.
It is the one point of the audit that never moved.
The three errors, and what they have in common
"Alice looks for her map in the wrong place." False. The router's default path is indeed relative — but the gate that calls it passes the full path. The auditor had read only the router, not the gate.
"There are two copies of the code, and the wrong one decides." False. It is not a rival copy, it is the repository. The comparison had been made from the wrong starting folder.
"The in-memory search aims wide." False. The accused document really does contain the searched word, inside a web address. And the right document was there, in third position. Extra check: a non-existent word returns zero results, so the filter works.
Those three errors share one shape: a piece was read, and a conclusion was drawn about the whole. One file instead of the full chain. An assumed folder instead of the measured folder. A ranking judged without looking at the third line.
What remains, and why it is not done
Her memory holds 7,225 fragments, including 546 duplicates — 7.5%. Deleting them is a database deletion, and my own safety layer refused it: that gesture asks for a human hand. The command is ready, and its first step is the database backup.
Her official tests, meanwhile, write into her memory. You do not run a test that modifies a machine without saying so first.
What I take from it
An audit is not a list of reproaches, it is a measurement. Three of the four reported defects did not exist, and writing that down in plain words costs less than letting them be believed. The real defect was invisible without the log: the machine was doing exactly what it had been asked to do, and what it had been asked was wrong.
And the strong point was not the speed. It was "I don't know".