· 2 min read
Your RAG demo is lying to you
Three months of shipping retrieval systems taught me that the failure you can't see is the one that costs you the customer.
Reactions
The first demo of a retrieval system is a magic trick. The thousandth user query is a probability distribution. Somewhere between the two, someone builds trust on answers that are confidently, fluently wrong.
The three failures nobody tests
In every engagement I've run — from NuronsAI client work to my own products — the same three failures were invisible until I built scoring to see them.
| Failure | How the demo hides it | How the harness catches it | | --- | --- | --- | | Unfaithful grounding | retrieval looks great on the easy questions | faithfulness scoring on golden answers | | Silent refusal | model answers around the missing context | refusal-correctness axis against "not in docs" | | Paraphrase fragility | the test set reads like the docs | same questions, rewritten adversarially |
The minimal stack that earns its keep
Not a platform. One file of golden pairs, a runner, and two gates that block merge. The full shape lives on the RAG Eval Harness project page:
assert run.faithfulness >= 0.92, "answers left the documents"
assert run.regression_count == 0, "last month was better"
What I stopped doing
- Tuning prompts against a favorite question. One question overfit is worse than no tuning; it teaches the model to be wrong with confidence.
- Retrieval metrics alone. Top-k accuracy measures the library, not the answer the user receives.
- Treating eval as a phase. The harness has to be running before the first bug ships, not scheduled after it.
The bottom line
If you can't tell me the faithfulness distribution of your last release, your demo isn't evidence — it's marketing. The good news: this problem is small, boring, and solvable in a week. That week is always cheaper than the support ticket where a confident wrong answer becomes a public one.
If your team would rather not spend that week, reach out.
