A green board over a leaving user
Is the assistant healthy? — check the eval dashboard.
faithfulness 0.94 ✓ · answer-relevance 0.91 ✓ · context-precision 0.89 ✓ · toxicity 0.00 ✓ — all green, shipping daily. Meanwhile, in production: 38% of users rephrase the same question twice, then close the tab.
Every light is green. The team ships with confidence. And retention slides, week over week, with no failing test to point at. The dashboard says the patient is healthy; the patient is leaving. Both are telling the truth — which is the whole problem.
This is the capstone, and here is why it had to be last. Five times now, every teardown ended the same way: instrument it. Build a standing check. Trust the trace. You have spent the series learning to build instruments and believe them. So the final lie is the one you’d never think to check — the instrument itself. What happens when the thing measuring health is the thing that’s broken? You don’t even get a red light to chase. You get silence, and churn.
The reflex is to add more eval cases. You can’t add your way out of measuring the wrong thing.
Why it happens
A passing eval is a claim, and you have spent five teardowns learning that claims must be checked against reality — even your own.
Every other teardown localized a fault inside the pipeline: a stage lied, you found it. The capstone inverts the move. Run the full diagnostic on the system and every stage passes — retrieval is faithful, generation is grounded, nothing is toxic. The system is fine against the questions you asked it. The fault isn’t in the patient. It’s in the thermometer.
An eval has two hidden assumptions, and a green board is only as honest as both. First, the inputs: your eval queries must resemble what users actually type — including the messy, ambiguous, half-formed ones that precede a rage-quit. Second, the metric: it must measure whether the user got what they needed, not whether the model was faithful to whatever it retrieved. Faithfulness can be 1.0 on an answer that is perfectly grounded in the wrong document and completely useless to the human.
So the capstone’s question isn’t “which stage broke?” It’s one rung higher: is the instrument measuring what users actually experience? The only way to answer it is the move this whole series has drilled — check the claim against an independent source of truth. Here the source of truth is the churn.
Where does it break?
A bad answer splits one way: the model either never got the answer-bearing context, or itgot it and still failed. Answer that, and you've halved the search space. Then localize within the failing half.
- ingest
- chunk
- embed
- retrieve
- rerank
- assemble
- prompt
- generate
- post
Descend the pipeline
Run the diagnostic and the pipeline passes everywhere — so the fault is in the measurement layer. Descend it: take the instrument apart on its own two assumptions, inputs and metric, and check each against the churn.
generate✓ cleared Is the pipeline itself actually broken on the eval set?
Replay the entire eval suite. Retrieval returns relevant context, the model answers faithfully to it, nothing is unsafe. Against the questions in the eval set, the system genuinely performs. There is no broken stage to find. That’s not the system passing the test — it’s the test passing the system. Keep descending, into the instrument.
post✕ fault here What does the eval measure, and on which inputs — and does either resemble a churning user?
Open the harness. Two findings, and they compound:
The 200 eval queries are clean, well-formed, single-intent — generated from the docs themselves. Real users ask: “why is my thing broken” / “the sla one” / “does this cover the eu region?? or no”. The eval never sees the queries that precede a rage-quit.
The headline metric is faithfulness — is the answer grounded in the retrieved context? It says nothing about whether the user’s problem got solved. An answer can be 0.94 faithful to a confidently-retrieved wrong document.
The instrument measures fidelity-to-context on easy questions. Users are failing on hard questions in a way fidelity can’t see. The fault is the measurement stage — green because it’s asking the wrong thing of the wrong inputs.
calibrate against reality✓ cleared Cross-check — rebuild the eval from real churned sessions and score task success. Does the board finally tell the truth?
Pull 80 sessions that ended in rephrase-and-leave. Run them through the system and score task success — did the user get unblocked? — instead of faithfulness.
task-success on real churn sessions: 0.55. The board goes red — and for the first time it agrees with retention. Faithfulness on the same sessions is still 0.92: the model is loyally, fluently answering the wrong interpretation.
There it is. The system was never as healthy as the dashboard claimed; the dashboard was measuring the wrong thing on the wrong data. A red eval that matches reality is worth infinitely more than a green one that lies — and you only get it by checking the instrument against the one source of truth it can’t fake: what users actually do.
Flip the fix
Theory's cheap. Take the same broken system and flip the fix yourself — watch the trace change and the eval scores move.
Is the assistant healthy? (production: 38% rephrase-and-leave)
faithfulness 0.94 ✓ on 200 clean, doc-generated queries — all green, shipping.
38% of users rephrase the same question twice, then leave. The board never sees them.
The dashboard says: healthy. Reality says: users walking. The dashboard wins the meeting.
Make it a standing check
The fix is to rebuild the instrument from reality: seed the eval set from real user queries — especially the failure-shaped ones that precede churn — and measure task success / resolution, not just faithfulness. Then do the thing almost no one does: calibrate the offline eval against the online signal continuously, so when they drift apart, the eval is the thing you fix, not the thing you trust. An eval you never check against reality is just a hallucination with a CI badge.
And this is where the series resolves. You have now been lied to six times — by the model, by your own fix, by the agent, by the critic, by the citation, and finally by the instrument you built to catch the other five. One thing survived every round: the method. Show the failure. Build the mental model. Split and localize against the real trace. Walk the stages. Instrument it. Distrust, and check. It survives because it assumes nothing and verifies everything — including itself. That is the whole point. The hero was never a tool: not the bigger embedder, not the second agent, not the eval suite. Every component in your stack can fool you. The discipline can’t.
That is also exactly what the cohort is. The teardowns teach you to read a trace; the cohort is where I sit with you and we read yours — your broken system, your real failure, live. If you’ve felt any of these six in your own work, that’s the conversation worth having.
- Why do offline LLM evals pass while production metrics get worse?
- Because a passing eval is only as honest as its inputs and its metric. If the queries are clean and synthetic and the metric rewards fidelity to retrieved context, the system can score 0.9 while confidently answering the wrong interpretation of the messy questions real users ask. The board is green because it's blind.
- How do you build an eval set that reflects real users?
- Seed it from real sessions — especially the failure-shaped ones that precede churn, like rephrase-and-leave — and score task success or resolution, not just faithfulness. Then continuously calibrate the offline eval against an online signal, so when they diverge you fix the eval rather than trust it.
- When offline and online numbers disagree, which one do you trust?
- The online signal — actual user behaviour is the ground truth your offline eval is only trying to approximate. A divergence means the eval is mis-measuring; the fix is to make the offline metric and dataset predict the online outcome, not to dismiss what users are doing.