niym academy

Niym Academy · a teardown series by Shivam Aggarwal

Your AI system is lying to you. With a straight face.

The model swears it doesn’t know. The agent insists it’s making progress. The eval board is all green. Every one is a lie — and they climb the stack, each more sophisticated than the last, until the final one is told by the very instrument you trusted to catch the other five. Six teardowns, one descent. You’ll leave able to diagnose any of them — and trust nothing you haven’t checked.

Read teardown 01 ◢◣See the six

Six-part series · one teardown a week · all of it free · connect for hands-on help

The surface area

One method. Every layer of a modern AI system — not just RAG.

The usual way

Swap the model. Bump the prompt. Add a tool. Trust the green dashboard. Repeat — believing whatever the system tells you.

The diagnostic way

Assume every component can lie. Localize against the real trace. Verify. Instrument. Distrust-and-check — until exactly one stage owns the fault.

The method · the one thing that can’t lie

A repeatable diagnostic, not a bag of tips

Every component in your stack can fool you — the model, the embedder, the agent, the critic, the citation, the eval. The discipline can’t, because it assumes nothing and checks everything. Every teardown runs this same spine.

  1. 01The failureThe broken behaviour
  2. 02The modelWhy it happens
  3. 03Diagnostic flowWhere does it break?
  4. 04Walk the stagesDescend the pipeline
  5. 05Try itFlip the fix
  6. 06Instrument itMake it a standing check
The series

Six Lies Your AI System Tells You

Every one of these is your AI system lying to you with a straight face — and the lies climb the stack, each more sophisticated than the last, until the final one is told by the very instrument you trusted to catch the other five.

I

The data layer lies

The plumbing. The model is innocent — the defect sits four stations upstream. And you can’t even trust your own fixes; only the trace tells the truth.

  1. 01RAGYour RAG says “I don’t know” for a question the docs answerThe correct passage is sitting in the index, but it never reaches the model.read free →
  2. 02Vector modelsYou scaled the embedding model. Search got worse.You doubled the embedding dimension and recall went down, not up.read free →
II

The system’s behaviour lies

Control. We leave the static pipeline. The fault isn’t the data anymore — it’s the system’s judgement of its own progress.

  1. 03AgentsYour research agent loops on the same tool until it times outGiven a simple task, the agent calls search() thirty times and never returns.read free →
III

Your safeguards lie

Trust theatre. You bolt on a layer whose whole job is to catch errors — and it’s asleep. Every guardrail becomes a new place for the lie to hide.

  1. 04Multi-agentYour critic agent rubber-stamps the planner's hallucinationThe critic was supposed to catch errors. It congratulates the planner instead.read free →
  2. 05Doc generationYour report generator invents a number — with a citationAn authoritative-looking figure, complete with a footnote, that exists in no source.read free →
IV

Your instruments lie

The capstone. For five teardowns you learned to build instruments you can trust. Now the lens turns on the instruments themselves — what happens when the thing measuring health is the thing that’s broken?

  1. 06EvalsYour evals are all green and your users are leavingEvery metric passes in CI. Production satisfaction is quietly cratering.read free →

The payoffAfter six lies, one thing is left standing: the method. Localize against the real trace, verify, instrument, distrust-and-check. Every component can fool you — the model, the embedder, the agent, the critic, the citation, the eval. The discipline can’t, because it assumes nothing and checks everything. The hero was never a tool. It’s the method — and it’s exactly what I’ll help you build in your own system.

Who writes it

Shivam Aggarwal

Senior engineer · Gurgaon, India

10+ years shipping software, the last 4 deep in AI. Founding engineer and now Head of AI at an AI-native workflow-automation startup — where debugging AI systems in production isn’t a hobby, it’s the job. These are the failures I’ve actually chased down, and the method I use to do it.

I think in public, too — long-form on the failures most teams hit:

And I ship: merged fixes to the MCP Python SDK, LangWatch, and DocBrain (self-hosted RAG). More at niym.ai →

Work with me

The series teaches you to read your own traces. I’ll read yours.

Every teardown here is free — read all six. But if you’re staring at a version of one of these in your own system, or you want hands-on mentoring on your real stack, that’s the conversation worth having. No application, no waitlist — reach out directly and we’ll find a time.

Write me an email →Book a time slot →