The right metric is not the one on display

Response time is the metric everyone displays. On a server that does one thing at a time, it is also the one that lies most elegantly: it measures what the request experienced, not what everyone else waited.

1. The word that decides everything

Our server stores its data in an embedded database, in synchronous mode. That word means there is no queue and no parallelism: every database call blocks the program's single thread of execution, from start to finish.

As long as queries take under a millisecond, you never see it — and that is precisely the trap. Behaviour is excellent right up to the exact moment it becomes catastrophic, with no transition zone to warn you.

A single slow query does not slow its caller: it puts everyone on hold, including the page that tells you whether the service is healthy. That page will answer late, and for the same reason as the others — which makes it useless at the precise moment you need it.

2. What response time does not say

Take a request handled in two milliseconds. That is an excellent figure, and it is accurate. But it does not say how long it waited before being handled, nor how many others were waiting behind it while it held the thread.

Averages make the misunderstanding worse: a few very slow requests drowned in thousands of fast ones give a reassuring average. High percentiles — the value below which 95% or 99% of requests fall — fix part of the problem, but remain per-request measurements. And the trouble is not in the request. It is in the thread.

3. The measurement that actually answers

So we measure event-loop lag. The principle fits in a sentence: we ask the program to call us back in exactly ten milliseconds, and we look at how long it actually took to return.

If it comes back at ten, it was busy with nothing. If it comes back at forty, it spent thirty milliseconds elsewhere — and during those thirty milliseconds, no request was progressing. That figure depends on no caller. It describes the state of the machine, not the state of a client.

It is also the measurement that tells you what to fix. A high response time can come from the network, the client, a third-party service. High loop lag can only come from one thing: synchronous work you are inflicting on yourself.

4. The protections it led us to

The measurement shaped the defences, in this order: refuse fast beyond a certain number of in-flight requests, rather than growing an invisible queue; limit throughput per identity, to absorb normal bursts and cut hammering; and cache repeated reads for a few seconds, because an animated map polling the server sixty times a second asks the same question sixty times.

Under a burst of three hundred simultaneous requests, two hundred and twenty were refused by the rate limit, and none by saturation. That zero is the most useful figure in the set: it says the cheapest protection cuts first, and the blunt one stays in reserve. Without measuring the loop, we could neither have set that order nor checked that it held.

🔭