The requirement turns up in more and more tender documents: the system must be \u201cexplainable\u201d. Phrased that way, it cannot be verified — because it says nothing about who has to understand what.
Five different needs
The person affected by a decision wants to know why they got this result and what they could change to get a different one. They need an explanation in ordinary language, concrete and actionable.
The operator using the system wants to know how reliable this result is and when not to trust it. They need a confidence signal and a list of situations where the system is weak.
The team building it wants to understand why the system fails on a case, so they can fix it. They need technical inspection tools.
The lawyer or compliance officer wants to be able to demonstrate that the decision meets legal requirements. They need documentation of the process and traceability.
The external auditor wants to verify the organisation's claims independently. They need methodology, test data and the ability to reproduce results.
No single mechanism satisfies all five. A feature-importance chart, for instance, tells the affected person nothing useful.
Global and local explanations
A global explanation describes how the system works overall: which variables matter in general, what rules it learned. Useful for validation and documentation.
A local explanation describes why this decision was taken for this case. That is what the affected person and the operator need.
Many organisations deliver a global explanation where a local one was required, and consider the requirement met.
Counterfactual explanations
For the affected person, the most useful form is usually counterfactual: \u201cthe decision would have been different if X had been Y\u201d. It is concrete, checkable and points to an action.
It also has a limit that must be respected: a counterfactual explanation should name something the person can actually change. \u201cIf you had a different year of birth\u201d is not an explanation, it is an observation.
The trade-off with performance
It is often said that simpler models are more explainable but weaker. Sometimes true, but not always — on many tabular problems a simple model is comparable to a complex one.
The right question is not \u201chow much do we lose by choosing an explainable model\u201d, but \u201chave we checked whether we lose anything\u201d. Often the answer is no, and the complexity was chosen out of habit.
In practice
Before writing \u201cthe system must be explainable\u201d in a document, complete the sentence: who must understand what, when, and what will they do with the explanation. The rest follows, and the requirement becomes one that can be verified.