Bigger model, worse neighbours
quarterly churn cohort analysis
1. blog: ‘Churn’ (marketing) · 2. hr-handbook: ‘quarterly reviews’ · 3. recipes: ‘the Cohort’ (a cocktail) … the actual analysis sits at rank 312.
You did the responsible thing. The 384-dimension embedding model felt small, so you upgraded to a 1024-dimension one — newer, higher on the leaderboard, more expensive to run. You re-indexed all two million documents on a fat GPU. And search got worse. Queries that used to land now return word-association soup: a query for “quarterly churn cohort analysis” surfaces a marketing blog titled “Churn”, an HR doc about “quarterly reviews”, and a cocktail recipe called “the Cohort.” The one document that actually answers it — your Q3 retention analysis — fell to rank 312.
Every dashboard you’d think to check says the upgrade went fine: GPU utilisation, throughput, index size, p99 latency — all green. So the instinct is to blame the model. The new one must be bad. You’re one click from rolling back to the small model and calling it a win. Don’t. The model is innocent, and the trace is about to prove it.
Why it happens
Bigger isn’t better and smaller isn’t the fix. Only the trace tells you what the model actually did to your vectors.
Retrieval ranks documents by a similarity score computed over vectors. That score has two moving parts people collapse into one: the geometry (do two vectors point the same way — does the model think the texts mean the same thing?) and the metric (how your index turns two vectors into a single number it can sort by). Swap the model and you can silently change the geometry, the metric’s assumptions, or both.
When search degrades after a model change, the reflexive question is “is the new model worse?” — a question about capacity. Almost always it’s the wrong question. A more capable model with a miscalibrated metric doesn’t give you better neighbours; it gives you wronger neighbours, with more confidence. The defect isn’t the model’s intelligence. It’s the calibration between what the model emits and what your index assumes it emits.
So we don’t ask “is the big model dumb?” We ask the same first question as every retrieval failure: was the answer-bearing document retrieved at all? Then we localise.
Where does it break?
A bad answer splits one way: the model either never got the answer-bearing context, or itgot it and still failed. Answer that, and you've halved the search space. Then localize within the failing half.
- ingest
- chunk
- embed
- retrieve
- rerank
- assemble
- prompt
- generate
- post
Descend the pipeline
The tree puts us on the retrieval side — the answer doc never made the cut. Now descend it. Each stage is one measurement against the real vectors, until exactly one stage owns the fault.
chunk✓ cleared Did the answer document survive indexing intact, same as before the upgrade?
Nothing changed here. Same chunker, same boundaries, same 2M chunks. The Q3 churn analysis is in the index as a clean unit — we can fetch it by id and read it. So the document exists and is findable in principle. Chunking is cleared.
embed✕ fault here Do the vectors carry the meaning — and does the index’s metric match how the model scales them?
Here’s the smell, and it’s a single number. Pull the raw vectors and measure their norms (their length). The old model and the new model are not the same shape:
old 384-dim model · vector norms ≈ 1.00 (already unit-length)
new 1024-dim model · vector norms range 0.31 … 22.4 (wildly un-normalized)
That difference is the whole case. Your vector index scores with inner product (the fast default in most vector DBs). Inner product multiplies direction by magnitude. With the old model every vector was unit-length, so magnitude was a no-op and inner product behaved exactly like cosine — pure meaning. The new model emits vectors with norms spanning two orders of magnitude, so now a handful of high-magnitude “loud” vectors win every query, regardless of what they’re about.
Measure it directly. For the query vs. the document that truly answers it vs. the junk blog:
cosine(query, answer) = 0.81 — the meaning is right there. But inner-product(query, answer) = 4.2, because the answer doc’s norm is a modest 5.1.
cosine(query, blog) = 0.39 — barely related. Yet inner-product(query, blog) = 71.5, because that vector’s norm is a booming 22.4. Magnitude, not meaning, put it at rank 1.
The geometry is fine — cosine knows the answer doc is the closest thing in the index. The model did its job. It’s the metric, reading un-normalized vectors, that’s lying.
rerank✓ cleared Cross-check — re-score the same candidates by cosine. If the answer surfaces, the vectors were never the problem.
Take the top-200 the broken index returned and re-rank them with cosine similarity instead of inner product — same vectors, one different scoring function:
The Q3 churn analysis jumps from rank 312 to rank 1. The cocktail recipe drops off the page.
That’s the proof. The model’s embeddings carried the answer the entire time; only the inner-product metric, fed un-normalized vectors, buried it. The fault is the embedding stage — specifically the mismatch between the model’s output scale and the index’s scoring assumption. A recall failure born the instant you swapped models, fatal at retrieval, with a perfectly capable model taking the blame.
Flip the fix
Theory's cheap. Take the same broken system and flip the fix yourself — watch the trace change and the eval scores move.
quarterly churn cohort analysis
Churn is the silent killer of SaaS — here’s why retention matters for your bottom line.
The Cohort: a gin cocktail with elderflower, grapefruit, and a dry vermouth float.
Quarterly reviews: how we run the performance cycle at the end of every quarter.
Q3 churn cohort analysis: retention by signup-month cohort, the 90-day drop-off curve, and the three segments driving it.
Top hits are loud, off-topic vectors. The Q3 analysis is at rank 312.
Make it a standing check
Fixing the ranking is a one-liner — L2-normalize the vectors at index and query time (or set the index metric to cosine), which turns inner product back into pure-meaning scoring. Recall snaps back, and you keep the better model. But fixing this query is a bug fix. Closing the class is the teardown.
The failure mode is a calibration mismatch between an embedding model and its index — and it’s invisible to every infra dashboard, because nothing broke; the numbers just quietly stopped meaning what you assumed. So instrument the assumption itself:
- A unit test on the embedding step: assert every emitted vector has norm ≈ 1.0 (or that normalization runs). The day someone swaps to a model that emits un-normalized vectors, this goes red before a single bad result ships.
- A small recall@k eval set — 15 queries paired with the document that truly answers each — run in CI on every model swap and every index-config change. It catches metric/normalization regressions that a leaderboard score never will.
And note what the try-out taught you about your own instinct: rolling back to the small model appears to fix it — because that model happened to emit normalized vectors. It’s a coincidence dressed as a diagnosis, and it quietly downgrades your retrieval to dodge a one-line fix. You couldn’t trust the symptom in teardown 01. Here you learn you can’t even trust your own fix. Only the trace can be trusted — because it measures what is, not what you assumed.
That closes Act I: the data layer lies, and so does your intuition about it. Next we leave the static pipeline entirely — and meet a system that doesn’t just hold the wrong answer, but lies to you about whether it’s making any progress at all.
- Why would bigger embeddings make search worse instead of better?
- Capacity isn't the problem — calibration is. If the new model outputs vectors with widely varying magnitudes and your index scores by inner product, ranking is dominated by vector length rather than semantic direction. A more capable model just gives you confidently wrong neighbours until you normalize.
- Should I use cosine or inner-product (dot) similarity for my vector index?
- Use cosine — or L2-normalize and then use inner product — unless your embedding model is explicitly trained for raw dot product and emits unit-norm vectors. Inner product on un-normalized vectors lets magnitude hijack ranking. When in doubt, normalize and compare by cosine.
- What silently changes when you swap embedding models?
- Vector magnitude and normalization, the instruction prefixes the model expects (many need 'query:' and 'passage:'), the pooling strategy, and the dimensionality — any of which can quietly break retrieval even though throughput and latency look fine. Re-run a recall eval on every model swap.