Evaluating AI Agents: What We Get Wrong & How to Fix It
Registration
About Event
Evaluation has gone from scoring labels on fixed test sets, to probing knowledge and in-context learning in early GPT models, to tool use with ReAct, to agents acting in live environments. The models changed; our evaluation habits mostly didn’t. We still report one number for what is really a stochastic policy, shaped by sampling, tools, infrastructure and judges.
This talk follows that history to show where inherited assumptions fail for agents, and offers practical fixes for evaluations you can trust.