🤖🍨 Sundae Robotics 04: Engineering Robotic Simulators for Evaluation, Real-to-Sim Transfer & Robot Learning with Digital Twins
🤖🍨 Grab a sundae and join Sundae Robotics, a private, invite-only Sunday series bringing together robotics researchers, founders, and builders working at the frontier of physical intelligence.
Sundae Robotics 04
Engineering Robotic Simulators for Evaluation and Beyond
Featured Talk: Real-to-Sim, Digital Twins & Simulation for Robot Learning
Keynote: Kaifeng Zhang
Ph.D. Student, Columbia University · Research Intern, World Labs
Simulation has long been a critical tool in robotics, but building a useful simulator requires far more than creating realistic visuals or tuning a handful of physics parameters. A simulator must reproduce the parts of the real world that actually matter for robot behavior: appearance, dynamics, task structure, and the physical interactions that determine whether a policy succeeds or fails.
In this talk, Kaifeng will present an engineering perspective on how simulation can support different stages of the robot learning pipeline. He will first introduce Real2Sim-Eval, a framework for converting real-world scenes into simulatable environments for policy evaluation. By carefully matching both visual appearance and physical dynamics, Real2Sim-Eval makes simulation a more reliable proxy for real-world robot performance. The approach combines Gaussian Splatting to reduce the visual gap with learned digital twins to model real-world dynamics, allowing policies trained on real-world data to operate zero-shot in simulation and produce evaluation results that strongly correlate with real-world performance, including on challenging manipulation tasks involving deformable objects.
Kaifeng will then discuss ongoing efforts to extend simulation beyond evaluation toward data generation and policy training. These efforts explore how real-to-sim reconstruction can be automated through agentic workflows and how calibrated physics simulation can become a scalable data engine for robot learning. By combining automated environment reconstruction, accurate physical modeling, and sim-to-real transfer, this work points toward a future where simulation becomes a core component of the infrastructure used to train and evaluate robot foundation models.
Kaifeng's research spans robot manipulation, real-to-sim and sim-to-real transfer for policy learning, digital twins, and world models. His recent work includes Real2Sim-Eval (ICRA 2026), PhysTwin (ICCV 2025), PGND (RSS 2025), and GS-Dynamics (CoRL 2024). He is currently a Ph.D. student at Columbia University and a research intern at World Labs.
Pre-Reading
• Real2Sim-Eval
• PhysTwin
• PGND
• GS-Dynamics
Topics
• Real-to-sim reconstruction for robot policy evaluation
• Closing the visual gap with Gaussian Splatting
• Learned digital twins for modeling physical dynamics
• Simulation for deformable-object manipulation
• Automated real-to-sim pipelines with agentic workflows
• Calibrated simulation for data generation and policy training
• Sim-to-real transfer for robot foundation models
Open Discussion + Q&A
• What makes a simulator trustworthy enough for robot policy evaluation?
• How accurately do appearance and dynamics need to match the real world?
• Can real-to-sim reconstruction become fully automated?
• When does simulation provide more useful training data than real-world collection?
• How should calibrated physics and learned world models work together?
• What role will simulation play in building future robot foundation models?