🤖🍨 Sundae Robotics 06: World Models, Video Prediction & Robot Planning — V-JEPA 2

​🤖🍨 Grab a sundae and join Sundae Robotics, a private, invite-only Sunday series bringing together robotics researchers, founders, and builders working at the frontier of physical intelligence.

​Sundae Robotics 06
World Models, Video Prediction & Robot Planning
Featured Paper: V-JEPA 2 — Self-Supervised Learning for Understanding, Predicting & Planning in the Physical World

​Building World Models from Video: V-JEPA 2

​How do we build AI systems that understand physical reality, anticipate what happens next, and use those predictions to act in unfamiliar environments?

​This session explores V-JEPA 2, Meta’s self-supervised video world model for visual understanding, prediction, and robot planning.

​Rather than training a model to reconstruct every pixel of a video, V-JEPA 2 learns to predict abstract representations of how scenes evolve. The approach is designed to capture the structure of the physical world—objects, motion, interactions, and likely future states—without requiring large amounts of human annotation.

​V-JEPA 2 is pre-trained on large-scale natural video using self-supervised learning, allowing the model to bootstrap an understanding of physical dynamics from passive visual experience. Meta then adapts the model using a comparatively small amount of robot interaction data, enabling it to plan actions for previously unseen objects and environments.

​One particularly interesting result is zero-shot robot planning. Tasks can be specified using a goal image, and the model can reason over possible action sequences to perform behaviors such as reaching, grasping, and pick-and-place. Because the learned representation is largely task-agnostic, the system does not require a separate collection of demonstrations for every new task.

​The work points toward a broader vision of world models: AI systems that do more than recognize what is currently visible. They form internal representations of how the world behaves, predict possible futures, and use those predictions to choose actions.

​We’ll unpack the architecture, training recipe, experimental results, and implications of V-JEPA 2—and discuss how representation-space prediction compares with generative video models as a foundation for physical intelligence.

​Pre-Reading

​• V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
https://arxiv.org/abs/2506.09985

​• Meta AI — V-JEPA 2
https://ai.meta.com/research/vjepa/

​• Code Models
https://github.com/facebookresearchvjepa2

​Topics

• ​Self-supervised learning from large-scale natural video

• ​Joint Embedding Predictive Architectures and latent-space prediction

• ​Learning physical world models without reconstructing pixels

• ​Representations for motion, dynamics, and object interactions

• ​Predicting future actions and events from visual context

• ​Connecting visual world models with language models for reasoning

• ​Fine-tuning world models with limited robot interaction data

• ​Goal-conditioned planning using target images

• ​Zero-shot robotic reaching, grasping, and pick-and-place

• ​Learning task-agnostic representations for robot control

• ​World models versus generative video models

• ​Scaling physical intelligence from passive video

• ​Efficient alternatives to task-specific robot demonstrations

• ​The role of predictive representations in general-purpose embodied AI

​Architecture + Training

​V-JEPA 2 follows a two-stage recipe.

​First, an encoder and predictor are pre-trained through self-supervised learning on video. Instead of attempting to generate missing pixels directly, the system predicts representations of unseen or future portions of a video in a learned latent space.

​This encourages the model to focus on higher-level, predictable structure—such as objects, motion, geometry, and physical interactions—rather than spending capacity modeling every visual detail.

​Second, the pretrained world model is adapted using a relatively small amount of robot data. Meta reports training the robot component using 62 hours of data from the DROID dataset, after which the model can be deployed for planning in new environments.

​The result suggests a possible scaling strategy for robotics: learn broad physical priors from abundant internet-scale video, then connect those representations to action using much smaller quantities of expensive robot data.

​Why It Matters

​Today’s robotic learning systems often depend on large collections of demonstrations gathered for specific embodiments, environments, or tasks.

​V-JEPA 2 explores a different path.

​If models can acquire general physical understanding from passive video, robot data may become primarily a way of teaching a model how to act, rather than having to teach it the structure of the physical world from scratch.

​That raises a compelling possibility: increasingly capable robot systems whose understanding comes from large-scale observational learning, while relatively lightweight post-training connects that understanding to new embodiments and tasks.

​Open Discussion + Q&A

• ​What exactly does a useful “world model” need to represent?

• ​Is predicting latent representations fundamentally more scalable than generating future video pixels?

• ​How much physical knowledge can be learned from passive video alone?

• ​What information is lost when prediction happens in representation space instead of pixel space?

• ​Does strong visual prediction imply meaningful causal or physical understanding?

• ​How should we evaluate whether a world model actually understands dynamics rather than exploiting visual correlations?

• ​Can internet-scale video meaningfully reduce the amount of robot data required for embodied intelligence?

• ​How embodiment-agnostic can a useful world model really become?

• ​Are goal images a scalable interface for specifying robot tasks?

• ​What prevents this approach from extending from short-horizon manipulation to long-horizon planning?

• ​How should V-JEPA-style predictive models interact with vision-language-action models?

• ​Will future robotics stacks use one unified world model, or separate perception, dynamics, planning, and control models?

• ​When should we predict in pixels, tokens, object representations, 3D representations, or learned latent spaces?

• ​Could self-supervised world models become the physical-world equivalent of language-model pre-training?