A confident ‘I don’t know’
What's our SLA for P1 tickets?
I'm sorry — I couldn't find any information about the SLA for P1 tickets in the knowledge base.
This is the worst kind of wrong, because it looks humble. The bot didn’t hallucinate a number — it declined. A reasonable person reads that and concludes the SLA isn’t documented.
It is. sla-policy.md has been in the index since day one. It says, in plain text: “Priority 1
incidents carry a one-hour response and four-hour resolution service-level agreement.” The answer was
always there. The model just never saw it.
Why it happens
A wrong answer is a symptom. The defect is hiding in one stage of the pipeline — debugging is collapsing the distance between the two.
When a RAG system answers badly, the instinct is to blame the model — tweak the prompt, swap to a bigger LLM, raise the temperature, lower it. That’s treating the symptom.
A RAG answer is the output of a pipeline: chunk the docs, embed them, retrieve the nearest, maybe rerank, assemble a context window, prompt, generate. A failure anywhere upstream surfaces as the same thing at the end — a bad answer. The model is the last station on the line, so it takes the blame for defects introduced four stations back.
So we don’t ask “why did the model say that?” first. We ask “which stage broke?” — and we have a fast way to cut the search space in half.
Where does it break?
A bad answer splits one way: the model either never got the answer-bearing context, or itgot it and still failed. Answer that, and you've halved the search space. Then localize within the failing half.
- ingest
- chunk
- embed
- retrieve
- rerank
- assemble
- prompt
- generate
- post
Descend the pipeline
The tree points us at the retrieval side. Now descend it stage by stage — each stage is one yes/no question against the real trace — until exactly one stage fails.
chunk✓ cleared Does the answer survive chunking as an intact, retrievable unit?
The SLA policy is short. At the default 1,000-token chunk size it lands as a single clean chunk, sentence intact. Nothing is split across a boundary.
“Priority 1 (P1) incidents carry a one-hour response and four-hour resolution service-level agreement. Priority 2 incidents carry a four-hour response…”
The answer exists as a unit. Chunking is cleared — keep descending.
embed⚑ suspicious Does the question land near the passage in vector space?
Here’s the first smell. We embed the user’s question and the answer chunk and measure the cosine similarity directly:
cosine( 'What's our SLA for P1 tickets?' , answer_chunk ) = 0.41
0.41 is weak for a pair that’s semantically identical. The reason is vocabulary. The user wrote “SLA” and “P1”; the document spells them out as “service-level agreement” and “Priority 1.” The embedding model never expands the acronyms — it sees two different surface forms and places them far apart. The query is orbiting the wrong neighborhood.
retrieve✕ fault here Is the answer-bearing chunk in the top-k the model actually receives?
This is where it dies. Top-k = 5. Here’s what retrieval hands the model for this query:
“…new engineers are added to the on-call rotation after their first sprint. SLAs are reviewed quarterly…”
“SLA — Service Level Agreement. P1 — the highest incident priority…”
“When a P1 is declared, page the on-call lead and open a war room…”
Plausible neighbors, every one — they mention SLAs and P1s. But not one states the actual numbers. The chunk that does:
“Priority 1 (P1) incidents carry a one-hour response and four-hour resolution service-level agreement…”
Rank 47. Forty-two positions below the cutoff. The model was handed five chunks that gesture at the topic and never given the one with the answer. Given that context, “I don’t know” is the correct, faithful response. The model did its job. Retrieval is the fault.
generate✓ cleared Cross-check — given the right chunk, does the model answer correctly?
Before we commit, confirm it’s only retrieval. Paste the rank-47 chunk into the context by hand and ask again:
“P1 incidents carry a one-hour response and four-hour resolution SLA.”
Perfect answer. The generation side was never broken — it was starved. That’s the whole diagnosis: a recall failure born in the embedding stage, fatal at retrieval, with a healthy generator downstream.
Flip the fix
Theory's cheap. Take the same broken system and flip the fix yourself — watch the trace change and the eval scores move.
What's our SLA for P1 tickets?
New engineers join the on-call rotation after their first sprint. SLAs are reviewed quarterly by the platform team.
SLA — Service Level Agreement. P1 — the highest incident priority tier.
When a P1 is declared, page the on-call lead and open a war room within fifteen minutes.
Priority 1 (P1) incidents carry a one-hour response and four-hour resolution service-level agreement.
I'm sorry — I couldn't find any information about the SLA for P1 tickets in the knowledge base.
Make it a standing check
Fixing this one query is a bug fix. Making sure this class of failure can’t come back silently is the teardown.
The failure mode is a vocabulary gap: users speak in acronyms and shorthand, the docs speak in full terms, and the embedding space quietly separates them. There will be more — every team has its jargon. So we turn the fix into a standing check.
Build a small eval set of 12–15 questions phrased the way users actually ask them — acronyms, internal codenames, abbreviations — each paired with the document that truly answers it. Assert recall@5: the answer-bearing chunk must appear in the top 5 for every question. Wire it into CI so it runs on every re-index and every embedding-model change. The day someone swaps the embedder or adds a new acronym the expander doesn’t know, this test goes red — before a user gets a false “I don’t know.”
That’s the difference between patching a ticket and closing a failure class.
- How do I tell whether it's a retrieval failure or a generation failure?
- Ask one question: was the answer-bearing context actually retrieved into the model's prompt? If no, it's a retrieval-side failure (embedding, chunking, ranking). If yes but the answer is still wrong, it's generation-side. That single split halves your search space before you debug anything.
- How do you catch a vocabulary gap before it ships, without hand-maintaining every acronym?
- Build a small recall@k eval set of real, messy user phrasings — acronyms, shorthand, internal codenames — each paired with the document that truly answers it, and run it in CI on every re-index and embedding change. The day a new acronym slips the expander, the test goes red before a user hits a false 'I don't know'.
- When is query expansion the wrong fix, and you actually need a different embedding model?
- Expansion fixes surface-form mismatches like 'SLA' versus 'service-level agreement'. If recall stays low even when the query and document use identical words, the embedding model itself isn't separating the concepts — that's when you change or fine-tune the model, not the query.