Sundae Robotics 08: Dexterous Manipulation, Simulation & Precise Assembly
🤖🍨 Grab a sundae and join Sundae Robotics, a private, invite-only Sunday series bringing together robotics researchers, founders, and builders working at the frontier of physical intelligence.
Sundae Robotics 07
Dexterous Manipulation, Simulation & Sim-to-Real Learning
Featured Talk: Scaling Dexterous Manipulation with Simulation — From General Tool Use to Precise Assembly
Scaling Dexterous Manipulation with Simulation: From General Tool Use to Precise Assembly
Keynote: Tyler Lum
5th-year Computer Science PhD Student, Stanford University · The Movement Lab · Interactive Perception and Robot Learning Lab
Dexterous robot hands have the potential to perform a wide range of manipulation tasks, but learning policies that are both general and precise remains challenging. In this talk, Tyler will present two projects exploring how large-scale reinforcement learning in simulation can help bridge this gap.
First, Tyler will discuss SimToolReal, which learns dexterous tool-use policies in simulation that generalize across unseen tools and tasks and transfer zero-shot to the real world.
He will then present Play2Perfect, which studies how broad, task-agnostic play pretraining can provide reusable manipulation skills that are efficiently fine-tuned for precise, contact-rich assembly.
Together, these projects explore how simulation can be used to learn broadly capable dexterous behaviors while still enabling the precision and specialization required for challenging real-world tasks.
Tyler Lum is a 5th-year Computer Science PhD student at Stanford University studying artificial intelligence and robotics. He is advised by Prof. C. Karen Liu in The Movement Lab (TML) and Prof. Jeannette Bohg in the Interactive Perception and Robot Learning (IPRL) Lab, and is supported by an NSERC Postgraduate Scholarship (PGS-D).
Tyler's research focuses on building robots that can move and act in dynamic real-world environments in an elegant and efficient manner. He develops methods for integrating learning-based perception and control, and studies the inductive biases that can improve the efficiency and reliability of these systems.
His recent work focuses heavily on dexterous manipulation and sim-to-real robot learning. His projects include Play2Perfect, which studies dexterous play pretraining for precise assembly; SimToolReal, which develops object-centric policies for zero-shot dexterous tool manipulation; and Human2Sim2Robot, which explores sim-to-real reinforcement learning from a single human demonstration.
Before his PhD, Tyler studied Engineering Physics at the University of British Columbia (UBC), where he graduated as a Wesbrook Scholar. He worked with Prof. Michiel van de Panne on reinforcement learning and motion planning for quadruped robots and with Prof. Purang Abolmaesumi on deep learning for medical image analysis.
Tyler has also worked as a robotics research intern at RAI and as a Robotics R&D intern at NVIDIA, where he worked on reinforcement learning for humanoid loco-manipulation and dexterous robot control.
His broader research interests include intelligent robots, elegant motion, interactive perception, dexterous bimanual manipulation, reasoning through uncertainty, long-horizon planning, creative problem solving, common-sense reasoning, and exploration strategies.
Tyler's long-term goal is to create robots that can reason through uncertainty, continuously learn from their environments, adapt to new challenges, and become reliable enough to create tremendous value for people in their everyday lives.
Topics
• Scaling dexterous manipulation with large-scale reinforcement learning in simulation
• Learning policies that combine broad generalization with precise control
• Dexterous tool use across unseen tools and tasks
• SimToolReal and zero-shot transfer from simulation to the real world
• Object-centric policies for dexterous manipulation
• Sim-to-real reinforcement learning for multi-fingered robot hands
• Broad, task-agnostic play pretraining
• Learning reusable manipulation skills through simulated interaction
• Play2Perfect and efficient downstream fine-tuning
• Precise, contact-rich robotic assembly
• Bridging general-purpose manipulation and specialized precision
• Learning-based perception and control for dynamic real-world environments
• Inductive biases for more efficient and reliable robot learning
• Scaling robot learning through simulated experience
• Building dexterous systems that are both general and precise
Open Discussion + Q&A
• How can dexterous policies remain general across tools and tasks without sacrificing precision?
• What aspects of dexterous manipulation are hardest to transfer from simulation to the real world?
• How much simulation diversity is required for robust zero-shot transfer?
• What representations or inductive biases enable generalization across unseen tools?
• Can a single policy learn reusable notions of grasping, tool use, and contact-rich interaction?
• What does task-agnostic play teach a robot that task-specific demonstrations do not?
• How should broad manipulation skills be fine-tuned for tasks requiring extremely precise control?
• What makes contact-rich assembly fundamentally different from general object manipulation?
• How much real-world data is still necessary after large-scale simulation pretraining?
• Can play pretraining become a general foundation for dexterous manipulation?
• When does reinforcement learning in simulation outperform imitation learning from real demonstrations?
• How should simulation environments be designed to encourage transferable rather than simulator-specific behaviors?
• Can the same pretrained policy support both open-ended tool use and highly constrained assembly?
• How should learning-based perception and control be integrated for reliable dexterous manipulation?
• What would it take to scale simulated dexterous learning toward truly general-purpose robot hands?