Rethinking Evaluation in the Agent Era through Claw-Eval

​"After evaluating 300 agent tasks, we revisited two fundamental questions: what should we evaluate, and how can we make the evaluation trustworthy?"

​NICE Talk NO.176 We invite Bowen Ye, PhD student at Peking University, to introduce Claw-Eval, an end-to-end evaluation framework for LLM Agents.

​TIME:

​- Beijing Time: May 30, 10:00–11:00

​- Eastern Time: May 29, 22:00–23:00

​🤔 As LLMs evolve from chatbots into agents that can use tools and coordinate systems, evaluation is shifting from “Can it answer?” to “Can it reliably get things done?”

​Claw-Eval includes 300 human-validated tasks across general workflows, multimodal settings, and multi-turn conversations, with systematic evaluation on 14 frontier models.

​The talk will cover:

​🔹 Why final-output-only evaluation is not enough

​🔹 How full trajectory auditing reveals hidden failure modes

​🔹 What safety and robustness look like inside real workflows

​🔹 Key findings from evaluating 14 frontier models

​🔹 Open questions on what Agent evaluation should measure nextJoin us to rethink what it means to evaluate agents in the LLM era.