Learning objectives
- Understand why evaluation is essential
- Recognize that different tasks require different metrics
- Connect technical performance to practical usefulness
Introduction
Training a model is not enough. You also need to know how well it performs, whether it is reliable, and whether its behavior is acceptable for the intended use. Evaluation answers these questions.
A useful model is not simply the one with the highest number on a chart. In real applications, performance must be interpreted in context. The cost of errors, fairness concerns, speed, explainability, and robustness can all matter.
A strong AI practitioner always asks: How are we measuring success, and does that measurement match the real goal?
Why evaluation cannot be optional
Without evaluation, you do not know if the model is better than a baseline, stable across different cases, or safe enough for deployment. A model may sound impressive in a demo but fail under realistic conditions.
Evaluation also helps compare model versions, justify deployment decisions, and identify weaknesses that require more data or design changes.
Metrics depend on the task
Classification, regression, ranking, recommendation, and generative AI tasks all need different ways of judging quality. Even within classification, accuracy may be insufficient when classes are imbalanced or certain errors are costly.
A hospital, for example, may care more about missing dangerous cases than about keeping a simple overall accuracy score high.
Practical evaluation beyond metrics
Model evaluation often includes human review, edge-case testing, fairness checks, latency measurement, and monitoring after deployment. Technical scores alone rarely tell the full story.
A model that is accurate but too slow, too opaque, or too biased may still be unacceptable.
Examples
Spam filter
A spam filter should be tested not only for overall accuracy but also for how often it wrongly blocks important legitimate emails.
Loan approval support
An AI scoring tool must be evaluated for fairness across groups, not just for prediction performance.
Customer chatbot
A chatbot should be reviewed for helpfulness, factual accuracy, tone, and escalation behavior when it is uncertain.
Exercises
- Why is evaluation just as important as training?
- Describe one case where accuracy alone is not enough.
- What non-metric factors may affect whether a model is useful?
- Choose a real AI product and suggest three ways to evaluate it.
- Why should evaluation continue even after deployment?
Key takeaway
Model evaluation is about more than numbers; it is about whether the system performs well, behaves responsibly, and fits the real-world purpose.