Veritas·RAG
Open-source RAG evaluation framework: three retrieval strategies on one corpus, scored by an LLM judge, with regression detection between runs.
Problem: Evaluating RAG in Production
Production RAG systems are hard to evaluate. Deploying an optimization looks good in testing but degrades performance on real-world queries. Most teams rely on manual evaluation or naive accuracy metrics that miss latency, token cost, and hallucination risks.
Solution: Evaluation Pipeline
How it works
Ingestion layer: parsers for PDF, DOCX, PPTX, HTML, email and scanned documents (OCR), chunking that cuts at headings first and never splits a table row, and access-control metadata on every chunk. Re-uploading an unchanged file is idempotent.
Retrieval layer: three interchangeable strategies.
- Classic: hybrid search (dense in Qdrant + BM25, RRF fusion), then cross-encoder reranking; BM25 fallback if the reranker fails, reported in every answer.
- Graph: walks entity triples extracted from the corpus by Claude.
- Agentic: Claude drives search, fetch and finish tools in a loop.
- All three implement one interface (Strategy design pattern) and return the same result shape, so they are swapped by configuration and compared question by question.
Evaluation
- LLM judge (Claude): faithfulness, answer relevance, context precision and recall, and answer correctness when a reference answer exists.
- Retrieval metrics: precision, recall, hit rate, MRR.
- Regression detection between runs of the same configuration.
- Golden question sets in English and French.
Built like a product
- GDPR features: PII masking, erasure, export and retention, audit log, OIDC SSO and access control.
- Observability: Langfuse tracing (spans for generation and tool calls, per strategy), Prometheus and Grafana.
- React interface: Ask, Compare, Advisor, Eval and Regressions views, in English and French.
- CI/CD with GitHub Actions: ruff, mypy, pytest with coverage, pip-audit, front-end lint, type-check, tests and build, Docker image builds.
Results: measured baseline
34 reference questions on the bundled corpus (41 documents, 107 chunks), measured on 2026-09-08 on the dense-only pipeline, right after the reranker fix (see below). A new baseline on the hybrid pipeline is in progress.
| Metric | Classic | Graph | Agentic |
|---|---|---|---|
| Retrieval recall | 0.765 | 0.765 | 0.544 |
| MRR | 0.554 | 0.554 | 0.356 |
| Faithfulness | 0.507 | 0.519 | 0.249 |
| Answer relevance | 0.862 | 0.854 | 0.521 |
| Latency per question | 21.7 s | 15.2 s | 18.0 s |
| Tokens (total) | 389k | 397k | 727k |
- On this corpus, graph retrieval matched classic retrieval quality at 30% lower latency.
- The agentic strategy used 1.9× the tokens for lower retrieval quality, and declined to answer 6 questions.
- No strategy wins everywhere: the point of the tool is to measure on your own data before shipping.
What evaluation exposed
The first LLM-judge scores were low while every request returned HTTP 200. Tracing the pipeline, I found a wrong model ID: the cross-encoder never loaded, and the system silently fell back to BM25. In the same pass: an index holding the corpus twice. After the fix, every answer reports which reranker actually ran, and the model is part of the configuration the regression check compares, so a degraded run is never compared with a healthy one.
Technology stack
Python
FastAPI
Pydantic
Claude APIQdrant
sentence-transformersBM25
PostgreSQL
React
TypeScript
Docker
GitHub ActionsLangfuse
Prometheus
Grafana