The field produces spectacular claims daily. A few questions applied quickly remove most of the noise.
1. Measured on what?
\u201cExceeds human performance\u201d means nothing without the task, the test set and the measurement method. The same claim can be true on a narrow benchmark and entirely false in real use.
2. Compared with what?
A result without a reference cannot be interpreted. Compared with a previous method? With a human? What kind of human, under what conditions, with how much time?
3. Who did the measuring?
A result published by whoever sells the product is not invalid, but it needs independent verification. The question is not whether they are lying, but whether anyone reproduced it.
4. Was the test visible during training?
If the test set was public before the model was trained, the model may have seen it. The result then measures memorisation, not capability. This is one of the most common problems with reported results.
5. What is not being said?
Error rate on hard cases, behaviour across subgroups, running cost, latency, what happened in the attempts that failed. The absence of this information is itself information.
6. Is it a demo or a system?
A demo works on the cases chosen for the demo. A system works on whatever turns up. The distance between them is usually measured in years.
7. Who should have contradicted it?
If a result would fundamentally change a field, specialists in that field would react. Their silence, or their reserved reactions, says more than the headline.
A quick test
Rephrase the claim as a verifiable prediction. \u201cIt will replace programmers\u201d becomes: which concrete task, done today by a programmer, will be done entirely by the system, without checking, and in what timeframe?
If the claim cannot be rephrased that way, it was not a claim about reality. It was a position.