A model can score well in isolation and still make the surrounding operation worse. The missing measure is often the decision the workflow exists to support.
A procurement assistant might extract fields accurately but still route exceptions incorrectly. A forecasting model might reduce average error while failing on the items where shortages are most costly. Aggregate quality hides operational asymmetry.
What good engineering looks like
Start evaluation from the task, decision and consequence. Build a representative test set with normal work, edge cases, stale data, conflicting evidence and prohibited actions. Compare the complete workflow against the existing baseline.
-
Decision correctness and confidence calibration.
-
Time to resolution and human review effort.
-
Cost of false positives and false negatives.
-
Performance on critical segments and exceptions.
-
Containment when data or tools are unavailable.
A practical starting point
-
Name the business decision in one sentence.
-
Collect examples from real operating conditions.
-
Set thresholds by risk class rather than one average score.
-
Retest after model, prompt, tool or data changes.
The decision to make
The best evaluation asks whether the organisation makes a better, safer and more timely decision with the system in place.