The challenge
The system runs on Graphiti and FalkorDB in Python. It has to give the current value of a business fact, give its value as of a past date, and say “no match” when the records don’t hold the answer. Those are exactly the behaviours that are easy to overstate in a test report.
The results could not be treated as independent. The same developer had built the system and every part of the evaluation. The code had been tuned against its own 20-record test set, and test data was visible in the extraction prompts. The scorer only checked that the expected value appeared somewhere in an answer, never that nothing wrong came with it, so a wrong answer could still pass. Extraction was not repeatable either: the same records and code produced different claims on each run.
There was a practical constraint too. The model and graph database ran only on the developer’s machine, and we had no access to the client’s systems, so all testing had to be designed to run offline or by the client.
- One person wrote the system, the tests, the answer key and the scorer
- The scorer rewarded answers that contained the right value alongside wrong ones
- Test data had leaked into the extraction prompts
- No direct access to the system under test
What I built
I wrote a blind set of 20 invented business records and 32 questions, with a frozen, SHA-256 hashed answer key. The questions cover current values, superseded values, as-of dates and cases where the right answer is “no match”. Each question carries expected and forbidden tokens and abstain rules. Two independent blind reviewers confirmed that every question had one clear answer. The set went to the client only, encrypted with AES-256 and with checksums for every file, so the developer never saw it.
The client ran the set three times and I scored the results against the frozen key, both with an automatic scorer and by hand. The two gradings agreed on every question across all three runs. The client received a per-question breakdown and a list of fabricated answers.
I then read the scorer line by line, found 42 weaknesses and turned them into 353 canary tests, 68 of them controls. They cover wrong values, near misses, superseded values, negation flips, entity swaps, Unicode and formatting tricks, number words and abstain-on-error. I checked the hashes of the client’s real scorer files before running the canaries against them. Each known scorer bug is recorded as an expected failure, so the suite flags it the moment it is fixed.
Every scorer fix I recommended was tested on patched copies first, and I kept only the fixes that removed errors without breaking correct answers. I also regraded the developer’s own counted runs strictly against their own answer key, which showed the reported totals were inflated. Everything runs on Windows and in a Linux Docker container that matches the client’s machine.
Architecture and stack
- Audit tooling
- Python 3.12 · pytest · Pydantic · Docker (python:3.12-slim)
- Integrity
- SHA-256 manifests · Frozen hashed answer key · 7-Zip AES-256 encryption
- System under test
- Graphiti · FalkorDB · Ollama (gpt-oss:120b) · nomic-embed-text
Outcome
The client now has an evaluation it can trust that does not depend on the developer marking their own work. The tested fix list cut false passes on the canary set from 153 to 15 and false fails from 70 to 7, and the canary suite stays in place so the same scorer bugs cannot quietly return.
The developer received a failure summary that explains the root causes without revealing the test set, so they could fix the system without being able to tune it to the questions. This is the same separation I recommend for any team evaluating an LLM or agent: the people who build the system should not also own the answer key.
- Blind test set with a frozen, hashed answer key, delivered encrypted
- Automatic and manual grading agreed on every question across three runs
- 353 canary tests, including 68 controls
- False passes on the canary set cut from 153 to 15, false fails from 70 to 7
Last updated