Reinforcement Learning with Function Approximation Done Right

​Why do many reinforcement-learning methods slip backwards after seeming to improve? The culprit is "policy iteration"—the bedrock of mainstream RL algorithms. Once we add function approximation, its built-in monotonic improvement guarantee breaks, and policies can ping-pong between mediocre performance. 

​However, this wasn't always the case. From the 1950s to the 1980s, researchers introduced “relaxed” versions of policy iteration that kept the progress monotonic. Building on those ideas, I will present a radically new policy-evaluation scheme that restores the classic guarantees even with general function approximation. I’ll close with some initial simulation results that showcase our method's promise. 

​This talk draws on our recent preprint, “Reliable Critics: Monotonic Improvement and Convergence Guarantees for Reinforcement Learning,” co-authored with Eshwar S. R., Aditya Gopalan, and Gal Dalal (https://arxiv.org/pdf/2506.07134).  

​About the Speaker: 
Gugan Thoppe
x: https://x.com/GThoppe
Current: Assistant Professor at the Indian Institute of Science, Bengaluru