What you can say about a tool nobody has used yet

We built a tool that turns a job posting into training. It runs, it holds up under load, it is confined. And nobody has yet used it to land a job. Here is what we can claim, what we cannot, and why the difference matters more than the rest.

1. What the tool does

You paste the text of a job posting. The tool extracts the skills actually being asked for, builds a path, and questions you on it — multiple choice, order-of-magnitude estimates, architecture trade-offs, storytelling drills. Every answer comes with a why. What you get wrong comes back sooner; what you master spaces out. A skill held long enough becomes a quarter of a coat of arms.

The hand-written exercise bank covers twelve skills across forty-six exercises: cloud, containers, declarative infrastructure, agents, retrieval-augmented generation, cost, security, data, model deployment, and how a US interview actually unfolds. For everything else — a sales role, a legal role, a trade we never anticipated — the tool can generate exercises, provided the user supplies their own API key. Without a key, the hand-written bank remains.

2. What we measured: load

The server stores its data in synchronous SQLite. That word decides everything: each query blocks the event loop. No queue, no parallelism. A single slow query puts everyone on hold, including the page that reports whether the service is healthy.

Normally you never see it — queries take under a millisecond. But the interface includes an animated map that polls the server in a loop. A tab left open overnight, or ten people at once, and you discover the limit the unpleasant way.

So we fired three hundred simultaneous requests at the live instance. The result:

The most instructive figure is the zero. It says the two protections are in the right order: the cheap one cuts first, and the blunt one stays in reserve. Had we seen the reverse, the fix would have been to revisit the settings — not to add a third protection.

3. What we measured: confinement

The first version ran directly on the machine, under the admin account. That account can become superuser without a password and drives the containers. In other words: a flaw in a multiple-choice exercise handed over the whole machine.

So the application got its own image, and the container that comes out of it has almost nothing: unprivileged user, read-only filesystem, all kernel capabilities dropped, no way to regain any, 512 MB of memory, 256 processes, no published port — it is reachable only through the front server — and no access to the container driver. The only place it can write is the volume holding its database.

The question we should have asked far sooner is not "how many requests does it hold", it is "if this process falls, how far does the attacker get". The two sound adjacent. They do not lead to the same work.

4. The defect that taught us most

The tool had to recognize skills in the text of a posting. The first version simply looked for a substring. 

// What the first version did
if (text.toLowerCase().includes(pattern)) { ... }

On a job posting for an AI architect, it reported an industrial asset management skill. The pattern was eam — and it sits inside team, a word that appeared six times in the ad. Later, the same defect filed a "commercial negotiation" posting under intelligent agents, because in French ia sits inside négociation. 

// What it does now: a word, not a fragment of a word
const re = new RegExp('(?<![a-z0-9])' + escape(pattern) + '(?![a-z0-9])', 'i')
return re.test(text)

This defect is worth telling because it never crashed. It raised no error and filled no log. It produced a plausible result, politely, every time. That is the most expensive category of defect: the one that asks for nothing, and that you only find by looking closely at the output, with a posting whose right answer you already know.

5. What we did not measure

Everything above is true and checkable. None of it says whether the tool is any use.

Nobody has yet used it to prepare a real interview. No candidate has been hired, or turned down, after training on it. We do not know whether spaced repetition holds up over three weeks once motivation drops. We do not know whether the hand-written exercises resemble the questions actually asked. We do not know whether the exercises generated for a trade we never anticipated are worth anything at all.

The only measurement that counts would be: someone trained on it, and it changed the outcome of their interview. We do not have it. We will not have it for weeks, and it will take more than one person before the number means anything.

We could have written this article without this paragraph. It would have been more flattering and just as accurate. It would merely have let you believe that a 10 ms 99th-percentile latency proves a tool helps someone — and those two things have nothing to do with each other.

6. What we take from it

🔭