Sundae Robotics 10: Scaling Robot Dexterity and Agility from Human Demonstrations
🤖🍨 Grab a sundae and join Sundae Robotics, a private, invite-only Sunday series bringing together robotics researchers, founders, and builders working at the frontier of physical intelligence.
Sundae Robotics 10
Human Demonstrations, Sim-to-Real RL & Dexterous Whole-Body Control
Featured Talk: Scaling Robot Dexterity and Agility from Human Demonstrations
Scaling Robot Dexterity and Agility from Human Demonstrations
Keynote: Lujie Yang
Member of Technical Staff, Amazon Frontier AI & Robotics · MIT EECS PhD
Human demonstrations provide a rich and expressive prior for teaching robots complex physical skills, from fine-grained tool use to agile whole-body loco-manipulation. However, translating them into executable robot skills is far from straightforward.
Differences in morphology, kinematics, and contact dynamics mean that direct pose retargeting frequently breaks critical contacts or violates physical constraints. Meanwhile, collecting large quantities of high-quality robot data through teleoperation remains costly and difficult to scale across new embodiments and environments.
In this talk, Lujie will present three complementary works that turn a handful of human demonstrations into physically grounded datasets and zero-shot hardware policies across bimanual arms, humanoids, and dexterous hands.
First, Lujie will introduce PhysicsGen, which integrates optimization-based kinematic retargeting with physics-driven trajectory optimization in simulation, transforming a few VR demonstrations into large-scale, physically consistent datasets that span different robot embodiments and physical parameters for training behavior cloning policies.
He will then present OmniRetarget, which scales data generation to humanoid whole-body loco-manipulation through an interaction-mesh formulation that explicitly preserves spatial and contact relationships between the robot, terrain, and manipulated objects. The method augments a single demonstration across terrains, object geometries, and humanoid embodiments for zero-shot sim-to-real RL transfer.
Finally, Lujie will discuss REGRIND, which brings this retargeting-to-RL paradigm to multi-fingered dexterity. It pairs interaction-preserving hand-object retargeting with residual reinforcement learning and careful system identification to achieve fluid, zero-shot tool use from a single demonstration, including tasks such as cutting with scissors and turning a screwdriver.
Together, these works explore how human demonstrations can serve as compact priors for generating large-scale, physically valid robot experience—allowing robots with very different embodiments to acquire dexterous and agile behaviors without requiring equally large-scale teleoperation datasets.
Lujie Yang is a Member of Technical Staff at Amazon Frontier AI & Robotics (FAR). He received his PhD from MIT EECS, where he was advised by Prof. Russ Tedrake, and his bachelor's degree from UC Berkeley, where he was advised by Prof. Francesco Borrelli.
Lujie's research aims to enable robots with human-like dexterity, agility, and intelligence in contact-rich manipulation and whole-body loco-manipulation. His work combines model-based optimization with data-driven techniques to create robust and data-efficient solutions for complex physical interaction.
His recent work spans dexterous manipulation, humanoid locomotion, whole-body interaction, reinforcement learning, trajectory optimization, and sim-to-real transfer. A recurring theme is using physical structure and optimization to transform limited human demonstrations into richer datasets and policies that can transfer across tasks, environments, and robot embodiments.
His work on OmniRetarget received both the Best Conference Paper Award and the Best Paper Award on Robot Manipulation and Locomotion at ICRA 2026. His PhysicsGen work on physics-driven data generation for contact-rich manipulation was published at RSS 2025.
Lujie has also worked on perceptive humanoid parkour, contact-rich manipulation planning, verified neural control, and optimization-based controller synthesis. Across these projects, his research explores how model-based knowledge and scalable learning can be combined rather than treated as competing approaches.
Topics
• Scaling robot dexterity and agility from human demonstrations
• Human demonstrations as priors for robot learning
• Bridging the human-robot embodiment gap
• Retargeting across different morphologies and kinematic structures
• Preserving physical contacts during motion retargeting
• Physics-driven trajectory optimization for robot data generation
• PhysicsGen and large-scale physically consistent synthetic datasets
• Training behavior cloning policies from augmented human demonstrations
• OmniRetarget and humanoid whole-body loco-manipulation
• Interaction-preserving retargeting through interaction meshes
• Preserving robot-object-terrain relationships across embodiments
• Augmenting demonstrations across terrain and object geometries
• Zero-shot sim-to-real reinforcement learning for humanoids
• REGRIND and multi-fingered dexterous manipulation
• Interaction-preserving hand-object retargeting
• Residual reinforcement learning for precise contact-rich behaviors
• System identification for robust sim-to-real dexterity
• Zero-shot tool use from a single human demonstration
• Combining model-based optimization with data-driven robot learning
• Scaling robot learning without scaling teleoperation at the same rate
Open Discussion + Q&A
• What information from a human demonstration actually needs to be preserved when transferring a skill to a robot?
• Why does direct pose retargeting fail for contact-rich manipulation and whole-body interaction?
• Should retargeting preserve joint configurations, end-effector trajectories, contacts, or higher-level interaction structure?
• How can physics optimization repair demonstrations that become infeasible on a different robot embodiment?
• How much synthetic variation can be generated from a single human demonstration before the original behavior stops being meaningful?
• Can physically grounded data augmentation replace large-scale robot teleoperation for some classes of tasks?
• What makes interaction meshes useful for transferring whole-body skills across humanoid embodiments?
• How can a single demonstration be generalized across different terrains, object geometries, and robot morphologies?
• What are the hardest parts of achieving zero-shot sim-to-real transfer for whole-body loco-manipulation?
• How should model-based optimization and reinforcement learning divide responsibility in complex physical tasks?
• Why is multi-fingered tool use particularly sensitive to small errors in contact geometry and dynamics?
• How much system identification is necessary for zero-shot dexterous sim-to-real transfer?
• Can residual RL reliably correct imperfections in retargeted human demonstrations without destroying the underlying skill?
• How far can the one-demonstration paradigm scale to genuinely new tasks rather than variations of an existing motion?
• Could human motion datasets eventually become a scalable pretraining source for general-purpose robot manipulation and locomotion?