RAG Eval Harness

The fastest way to make a RAG product worse is to make it look better. A hand-tuned prompt wins the demo, ships, and quietly degrades answers nobody was reading closely. This harness exists because I stopped trusting my own eyes.

The gauntlet

A golden set of 400+ question/answer pairs — real questions from real deployment, written by the humans who used the product — runs against every change on twelve axes. The two that gate merges:

def gate(run: EvalRun) -> bool:
    return (
        run.faithfulness >= 0.92          # answers stay grounded in retrieved context
        and run.answer_relevance >= 0.85  # answers address the question asked
        and run.regression_count == 0     # nothing that passed last week fails today
    )

What changed after the harness

| Before | After | | --- | --- | | "The demo looked good" | "0 regressions, faithfulness +0.03" | | Bug reports became arguments | Bug reports became golden questions | | Prompt edits were vibes | Prompt edits were PRs with scores attached | | Silent drift | Weekly scorecard history |

The lesson I keep re-learning

Evaluation is not a phase — it is the product's immune system. You design the body and the immune system together, or the first real-world week designs it for you, badly.

The harness is written as a small, boring Python package — the kind you can drop beside any retrieval stack. It's on GitHub once the first public cut lands; in the meantime, ask me about it and I'll walk you through the ugliest graphs.