How to Tell Whether Your AI System Is Actually Any Good
Most teams evaluate AI by trying it a few times and feeling impressed. Here is what measuring it properly looks like.
AI features are unusual in that they usually look good in a demo and fail in ways nobody notices. Traditional software either works or throws an error; a model produces a fluent, confident answer whether or not it is correct. Without measurement you have no idea which you are shipping.
Build the evaluation set first
Fifty real inputs with the expected outcome, collected before you build. Real means from actual users or actual documents, including the awkward ones. A set assembled from the cases your system handles well is a reassurance exercise, not a test.
Separate the two failures
- Retrieval: did the right source material come back? Measure this independently, because a generation failure and a retrieval failure look identical from the outside and require opposite fixes.
- Generation: given the right material, was the answer correct, complete and appropriately hedged?
Metrics that mislead
- User satisfaction alone. People rate fluent answers highly regardless of accuracy; confidence reads as competence.
- Average scores. One catastrophic answer that invents a policy matters more than fifty mediocre ones, and an average hides it perfectly.
- Benchmark performance of the underlying model. It tells you nothing about your documents, your users or your prompts.
- Volume of use. Adoption measures curiosity in month one and habit in month six; only the second means anything.
Catching regressions
Run the evaluation set on every prompt change and every model update. Providers deprecate and update models, and behaviour shifts without your code changing at all. Without an automated check, you discover this from a customer, which is the most expensive possible way to find out. It is also how we scope AI integration engagements, evaluation set first, model second.
- 50 real cases in an evaluation set
- 2 failure types, measured separately
- every prompt or model change re-evaluated
Frequently asked questions
Can a model evaluate another model's output?
Yes, and it scales well for a first pass — but it inherits blind spots and tends to reward fluency. Use it for breadth, and keep a human-reviewed subset as ground truth to check the judge itself.
How often should we re-evaluate?
On every change you make, and on a schedule regardless — monthly is reasonable. Model behaviour drifts underneath you, and the whole point of the set is that you find out before your users do.
More on this topic: Artificial Intelligence.
Keep reading
Want this built for your business? See what we do.