---
title: "RAG Eval Harness"
slug: rag-eval-harness
date: 2026-03-18
url: https://mohammadhameed.dev/projects/rag-eval-harness
status: published
tags: ["ai", "systems"]
technologies: ["Python", "pgvector", "PyTorch", "Docker"]
---

{/* Sample case study for the design refresh — repo link and metrics are
    placeholders until the harness is actually published. */}

The fastest way to make a RAG product worse is to make it look better. A
hand-tuned prompt wins the demo, ships, and quietly degrades answers nobody
was reading closely. This harness exists because I stopped trusting my own
eyes.

## The gauntlet

A golden set of 400+ question/answer pairs — real questions from real
deployment, written by the humans who used the product — runs against every
change on twelve axes. The two that gate merges:

```python
def gate(run: EvalRun) -> bool:
    return (
        run.faithfulness >= 0.92          # answers stay grounded in retrieved context
        and run.answer_relevance >= 0.85  # answers address the question asked
        and run.regression_count == 0     # nothing that passed last week fails today
    )
```

<Callout type="note">
Axes beyond the gates exist to keep us honest: citation precision, refusal
correctness, latency budget, retrieval stability under paraphrase.
</Callout>

<Figure src="/images/projects/rag-eval-harness-alt.svg" alt="Faithfulness against latency across runs" caption="Each dot is a run — the upper-left is where products live." />

## What changed after the harness

| Before | After |
| --- | --- |
| "The demo looked good" | "0 regressions, faithfulness +0.03" |
| Bug reports became arguments | Bug reports became golden questions |
| Prompt edits were vibes | Prompt edits were PRs with scores attached |
| Silent drift | Weekly scorecard history |

## The lesson I keep re-learning

Evaluation is not a phase — it is the product's immune system. You design
the body and the immune system together, or the first real-world week designs
it for you, badly.

The harness is written as a small, boring Python package — the kind you can
drop beside any retrieval stack. [It's on GitHub](https://github.com/MoeHameed/rag-eval-harness)
once the first public cut lands; in the meantime, [ask me about it](/contact)
and I'll walk you through the ugliest graphs.