Measuring what a system actually changed

Every product we ship carries a single number on its page. Not a dashboard, not a wall of charts — one figure that says what changed because the system exists. A clinic's wait time fell by a measurable amount. A bank reconciled at a measurable rate. The discipline is not in collecting the number; it is in choosing it honestly.

The temptation is to pick the flattering metric. Usage is up. Engagement is up. Something is always up. But a number is only credible when it could have gone the other way — when it measures the thing the client was actually trying to fix, and when we would have published it even if it had embarrassed us.

The metric has to be falsifiable

Before we build, we write down the number we expect to move and the direction we expect it to move in. If we cannot name it, we do not yet understand the problem well enough to start. This is the same instinct a scientist brings to an experiment: state the hypothesis first, then let the world disagree with you.

A number is only credible when it could have gone the other way.

When the system goes live, the number is no longer ours to spin. It is computed from the same records the client runs on, refreshed on the same cadence, visible on a public page. That is the part that makes people uncomfortable, and it is the part that makes them trust the rest.

A small example

Here is the query that backs the wait-time figure on the Clinic Engine page. It is deliberately boring — a median over a rolling window, computed where the data already lives.

select percentile_cont(0.5)
  within group (order by seen_at - arrived_at)
from visits
where arrived_at > now() - interval '30 days';

That is the whole trick. Publish the definition, compute it where the truth is, and let the figure stand or fall on its own. The number is not marketing. It is the smallest honest summary of a year of work.¹