Does Your Agent Know It's Lost? Uncertainty and Progress Signals for Reliable LLM Agents, with Sharon Li

​Does Your Agent Know It's Lost? Uncertainty and Progress Signals for Reliable LLM Agents
Talk by Sharon Li, Associate Professor, University of Wisconsin-Madison
Hosted by Beyza Ermis, Senior Research Scientist, Cohere Labs

​LLM agents are increasingly deployed on long-horizon tasks with tool use, irreversible actions, and unpredictable feedback. Yet we have few principled ways to tell, mid-episode, whether an agent is on track or quietly failing. Most uncertainty quantification (UQ) research still centers on single-turn QA, a poor match for interactive agents. In this talk, I'll present a general formulation of agent UQ and the challenges unique to agentic settings, from choosing uncertainty estimators to modeling how uncertainty evolves over an interaction. I'll then show that a powerful answer has been hiding in plain sight: RL post-training already yields an implicit step-level signal, the progress advantage, which recovers the optimal advantage function with no annotation or reward-model training. Across test-time scaling, UQ, and failure attribution, this free byproduct beats confidence baselines and even dedicated trained reward models.

This talk is part of Cohere Labs in Conversation, a limited series of talks, in which Cohere Labs scientists and engineers host external researchers for techincal talks and Q&A discussions on subjects related to our current explorations at Cohere Labs. We look forward to sharing these talks with you, giving you a glimpse into the problems we're exploring, and learning together from some of the greatest minds in the field.