← Six Lies Your AI System Tells You
Teardown 02 of 06Act I — The data layer liesVector models

You scaled the embedding model. Search got worse.

Symptom
You doubled the embedding dimension and recall went down, not up.
System
a semantic search index over 2M documents
Stages stressed
embedding · retrieval
Difficulty
advanced
§1The failure

Bigger model, worse neighbours

user ▸

quarterly churn cohort analysis

system ▸

1. blog: ‘Churn’ (marketing) · 2. hr-handbook: ‘quarterly reviews’ · 3. recipes: ‘the Cohort’ (a cocktail) … the actual analysis sits at rank 312.

✕ Bigger model · worse neighbours

You did the responsible thing. The 384-dimension embedding model felt small, so you upgraded to a 1024-dimension one — newer, higher on the leaderboard, more expensive to run. You re-indexed all two million documents on a fat GPU. And search got worse. Queries that used to land now return word-association soup: a query for “quarterly churn cohort analysis” surfaces a marketing blog titled “Churn”, an HR doc about “quarterly reviews”, and a cocktail recipe called “the Cohort.” The one document that actually answers it — your Q3 retention analysis — fell to rank 312.

Every dashboard you’d think to check says the upgrade went fine: GPU utilisation, throughput, index size, p99 latency — all green. So the instinct is to blame the model. The new one must be bad. You’re one click from rolling back to the small model and calling it a win. Don’t. The model is innocent, and the trace is about to prove it.

§2The model

Why it happens

Bigger isn’t better and smaller isn’t the fix. Only the trace tells you what the model actually did to your vectors.

Retrieval ranks documents by a similarity score computed over vectors. That score has two moving parts people collapse into one: the geometry (do two vectors point the same way — does the model think the texts mean the same thing?) and the metric (how your index turns two vectors into a single number it can sort by). Swap the model and you can silently change the geometry, the metric’s assumptions, or both.

When search degrades after a model change, the reflexive question is “is the new model worse?” — a question about capacity. Almost always it’s the wrong question. A more capable model with a miscalibrated metric doesn’t give you better neighbours; it gives you wronger neighbours, with more confidence. The defect isn’t the model’s intelligence. It’s the calibration between what the model emits and what your index assumes it emits.

So we don’t ask “is the big model dumb?” We ask the same first question as every retrieval failure: was the answer-bearing document retrieved at all? Then we localise.

§3Diagnostic flow

Where does it break?

A bad answer splits one way: the model either never got the answer-bearing context, or itgot it and still failed. Answer that, and you've halved the search space. Then localize within the failing half.

Diagnostic decision treeRoot question: Was the answer-bearing context retrieved? The retrieval side is the failure path; the failure localizes to the embedding stage.NOYES — still wrongWas the answer-bearingcontext retrieved?Retrieval sideanswer never reachedthe model · recall faultGeneration sidemodel had it andstill failed · reasoninglocalized →embed stage
Fig. 1 — decision tree · failure on the retrieval side, localized at embedding
§4Walk the stages

Descend the pipeline

The tree puts us on the retrieval side — the answer doc never made the cut. Now descend it. Each stage is one measurement against the real vectors, until exactly one stage owns the fault.

  1. chunk✓ cleared

    Did the answer document survive indexing intact, same as before the upgrade?

    Nothing changed here. Same chunker, same boundaries, same 2M chunks. The Q3 churn analysis is in the index as a clean unit — we can fetch it by id and read it. So the document exists and is findable in principle. Chunking is cleared.

  2. embed✕ fault here

    Do the vectors carry the meaning — and does the index’s metric match how the model scales them?

    Here’s the smell, and it’s a single number. Pull the raw vectors and measure their norms (their length). The old model and the new model are not the same shape:

    old 384-dim model · vector norms ≈ 1.00 (already unit-length)

    new 1024-dim model · vector norms range 0.31 … 22.4 (wildly un-normalized)

    That difference is the whole case. Your vector index scores with inner product (the fast default in most vector DBs). Inner product multiplies direction by magnitude. With the old model every vector was unit-length, so magnitude was a no-op and inner product behaved exactly like cosine — pure meaning. The new model emits vectors with norms spanning two orders of magnitude, so now a handful of high-magnitude “loud” vectors win every query, regardless of what they’re about.

    Measure it directly. For the query vs. the document that truly answers it vs. the junk blog:

    answer: q3-churn-analysis.md

    cosine(query, answer) = 0.81 — the meaning is right there. But inner-product(query, answer) = 4.2, because the answer doc’s norm is a modest 5.1.

    junk: blog/churn.md

    cosine(query, blog) = 0.39 — barely related. Yet inner-product(query, blog) = 71.5, because that vector’s norm is a booming 22.4. Magnitude, not meaning, put it at rank 1.

    The geometry is fine — cosine knows the answer doc is the closest thing in the index. The model did its job. It’s the metric, reading un-normalized vectors, that’s lying.

  3. rerank✓ cleared

    Cross-check — re-score the same candidates by cosine. If the answer surfaces, the vectors were never the problem.

    Take the top-200 the broken index returned and re-rank them with cosine similarity instead of inner product — same vectors, one different scoring function:

    re-ranked by cosine

    The Q3 churn analysis jumps from rank 312 to rank 1. The cocktail recipe drops off the page.

    That’s the proof. The model’s embeddings carried the answer the entire time; only the inner-product metric, fed un-normalized vectors, buried it. The fault is the embedding stage — specifically the mismatch between the model’s output scale and the index’s scoring assumption. A recall failure born the instant you swapped models, fatal at retrieval, with a perfectly capable model taking the blame.

§5Try it

Flip the fix

Theory's cheap. Take the same broken system and flip the fix yourself — watch the trace change and the eval scores move.

try-it · baseline (broken)4 results
Fixed query under test

quarterly churn cohort analysis

Embedding model
L2-normalize the vectors
Top results
blog/churn.mdrank #1score 71.50

Churn is the silent killer of SaaS — here’s why retention matters for your bottom line.

recipes/cohort.mdrank #2score 58.20

The Cohort: a gin cocktail with elderflower, grapefruit, and a dry vermouth float.

hr-handbook.mdrank #3score 44.00

Quarterly reviews: how we run the performance cycle at the end of every quarter.

q3-churn-analysis.mdrank #312score 4.20◂ answer-bearing

Q3 churn cohort analysis: retention by signup-month cohort, the 90-day drop-off curve, and the three segments driving it.

Answer

Top hits are loud, off-topic vectors. The Q3 analysis is at rank 312.

Evals
recall@100.00 / 0.90
neighbour_precision0.10 / 0.70
What changedThe new model emits vectors with norms up to 22×. Under inner-product scoring, magnitude — not meaning — wins. The geometry is fine; the metric is lying.
§6Instrument it

Make it a standing check

Fixing the ranking is a one-liner — L2-normalize the vectors at index and query time (or set the index metric to cosine), which turns inner product back into pure-meaning scoring. Recall snaps back, and you keep the better model. But fixing this query is a bug fix. Closing the class is the teardown.

The failure mode is a calibration mismatch between an embedding model and its index — and it’s invisible to every infra dashboard, because nothing broke; the numbers just quietly stopped meaning what you assumed. So instrument the assumption itself:

  1. A unit test on the embedding step: assert every emitted vector has norm ≈ 1.0 (or that normalization runs). The day someone swaps to a model that emits un-normalized vectors, this goes red before a single bad result ships.
  2. A small recall@k eval set — 15 queries paired with the document that truly answers each — run in CI on every model swap and every index-config change. It catches metric/normalization regressions that a leaderboard score never will.

And note what the try-out taught you about your own instinct: rolling back to the small model appears to fix it — because that model happened to emit normalized vectors. It’s a coincidence dressed as a diagnosis, and it quietly downgrades your retrieval to dodge a one-line fix. You couldn’t trust the symptom in teardown 01. Here you learn you can’t even trust your own fix. Only the trace can be trusted — because it measures what is, not what you assumed.

That closes Act I: the data layer lies, and so does your intuition about it. Next we leave the static pipeline entirely — and meet a system that doesn’t just hold the wrong answer, but lies to you about whether it’s making any progress at all.

Questions this raises
Why would bigger embeddings make search worse instead of better?
Capacity isn't the problem — calibration is. If the new model outputs vectors with widely varying magnitudes and your index scores by inner product, ranking is dominated by vector length rather than semantic direction. A more capable model just gives you confidently wrong neighbours until you normalize.
Should I use cosine or inner-product (dot) similarity for my vector index?
Use cosine — or L2-normalize and then use inner product — unless your embedding model is explicitly trained for raw dot product and emits unit-norm vectors. Inner product on un-normalized vectors lets magnitude hijack ranking. When in doubt, normalize and compare by cosine.
What silently changes when you swap embedding models?
Vector magnitude and normalization, the instruction prefixes the model expects (many need 'query:' and 'passage:'), the pooling strategy, and the dimensionality — any of which can quietly break retrieval even though throughput and latency look fine. Re-run a recall eval on every model swap.