Personal project · open source
Groundtruth — an evaluation gate for RAG
Built a harness that measures RAG changes before merge and blocks the PR when Recall@10 drops by more than 1 percentage point.
Problem
Changing chunking, search or the reranker “feels better” until someone measures it. Without a versioned baseline, a retrieval regression reaches production unannounced.
Available data
A golden set of 100 human-reviewed questions over one technical document (V1) and 18 approved questions over a three-document corpus (multi-document).
Architecture decisions
An objective gate
The baseline is versioned in the repo. A change fails if Recall@10 drops by more than 1 percentage point, with numeric tolerance so rounding does not cause failures.
Prove the test works
A deliberately broken retriever must bring the gate down. If it does not, the gate is useless.
Retrieval and answers measured separately
Recall, MRR and nDCG measure search; Faithfulness and Answer Relevancy (RAGAS) measure the answer. Either one alone is misleading.
Implementation
- GitHub Actions workflow that recomputes retrieval and applies the gate.
- Comparison of dense, hybrid and LLM-reranked search, at two chunk sizes.
- Multi-document: global search, a document-filter tool and graph expansion (GraphRAG), under controlled and tuned conditions, with cost, latency and tool calls per answer.
Result
- 0.88 → 0.98nDCG@10 with a reranker, Recall within the gate
- −2.78 ppRecall with GraphRAG vs. global search: the gate would block it
The reranker was recommended as the quality-first option. Graph expansion scored below global search on this sample — and the gate itself would have blocked the merge.
Limits and lessons
- The multi-document golden set is small (18 questions).
- Binary relevance cannot tell a direct answer from a passage that only adds context; graded relevance (0–3) would be the next step.
- Reranker cost and latency were not compared in V1.