\u201cIt is 94% accurate\u201d is the sentence that ends most project presentations and starts most of the trouble.
Why an average hides exactly what you care about
Accuracy is the proportion of correct predictions on a test set. It silently assumes that all errors cost the same and that all cases are equally frequent. Almost never true.
In a system that detects something rare — fraud, a defect, a disease — a model that always says \u201cno\u201d can reach very high accuracy while being of no use. And in a system where one kind of error costs far more than the other, averaging the two means nothing.
The questions that produce useful information
What are the two error types, and what does each cost? A false positive and a false negative have different consequences, sometimes by orders of magnitude. Evaluation must separate them rather than blend them into one number.
How does it behave across subgroups? The same average accuracy can hide good performance on the majority group and poor performance on groups less represented in training data. If you have not measured per subgroup, you do not know.
What happens at the edges? Performance on easy cases says nothing about hard ones. A test set built to be representative of difficulty is far more informative than one built at random.
How does the test data compare with reality? If the test set comes from the same period, source and distribution as training, the figure is optimistic. The test that matters uses data from a different period or a different source.
What happens when the input is odd? Missing data, unexpected formats, text in another language, extreme values. A system that degrades gracefully is a different thing from one that confidently produces nonsense.
Evaluation does not end at launch
Data distributions shift over time: processes change, people's behaviour changes, products change. A model stays fixed; reality does not.
That means evaluation has to be a continuous process, not an acceptance report:
- monitoring the distribution of inputs against training data;
- a sample of predictions reviewed by hand, periodically;
- a declared threshold below which the system stops or escalates to a human;
- a decision log, so a problem reported three months later can be investigated.
What an evaluation report should contain
A useful report states: what question was asked, on what data it was answered, which metrics were used and why those, how performance looks across subgroups, where the system errs systematically, what could not be measured, and what should be monitored after deployment.
A report containing one percentage and a chart is not an evaluation. It is an assertion.