

Théo Vincent - Gradient Iterated Temporal-Difference Learning
Talk Description: Temporal-difference (TD) learning is highly effective at controlling and evaluating an agent’s long-term outcomes. Most approaches in this paradigm implement a semi-gradient update to boost the learning speed, which consists of ignoring the gradient of the bootstrapped estimate. While popular, this type of update is prone to divergence, as Baird’s counterexample illustrates. Gradient TD methods were introduced to overcome this issue, but have not been widely used, potentially due to issues with learning speed compared to semi-gradient methods. Recently, iterated TD learning was developed to increase the learning speed of TD methods. For that, it learns a sequence of action-value functions in parallel, where each function is optimized to represent the application of the Bellman operator over the previous function in the sequence. While promising, this algorithm can be unstable due to its semi-gradient nature, as each function tracks a moving target. In this talk, we will modify iterated TD learning by computing the gradients over those moving targets, aiming to build a powerful gradient TD method that competes with semi-gradient methods. The evaluation reveals that this algorithm, called Gradient Iterated Temporal-Difference learning, has a competitive learning speed against semi-gradient methods across various benchmarks, including Atari games, a result that no prior work on gradient TD methods has demonstrated.
Bio: Théo Vincent is a Ph.D. student at TU Darmstadt and DFKI, the German Institute for Artificial Intelligence. He is working on temporal-difference learning methods, focusing on the off-policy setting. He works under the supervision of Jan Peters. Théo is currently interning at Meta FAIR, working on unsupervised learning in the team of Alessandra Lazaric.
Before his Ph.D., Théo graduated from ENS Paris-Saclay and Ponts ParisTech. He worked on computer vision problems in a Parisian lab, the Saint-Venant lab, and a Swedish start-up, Signality. He also spent some time at Harvard Medical School doing research in biostatistics.