When Agents Meet Physical Data: The Other Physics of Data Harnesses

​Ask an agent to find every night-time pedestrian frame across terabytes of dashcam video in S3. The first pass costs thousands of dollars and runs for hours. Once you have paid that cost, the agent's favorite move is to loop, inspect, and re-derive, which becomes the worst thing it can do.

​Most agent intuition was built for a different physics. Coding and business automation, where recompute is cheap, verification means re-running code, and state fits in context. That physics works. But like Newtonian mechanics at cosmic scale, it stops working when the data gets large enough.

​Physical AI data breaks every assumption. Video, sensor streams, logs from robots, vehicles, and factories. Petabytes never fit in a context window. Just verify it may mean another expensive inference pass. A follow-up question should recall what the system already paid to discover, not trigger a full perception run.

​In this session, Dmitry Petrov, creator of DVC and now working on Datachain, walks through what inverts when recompute is the enemy. Materialization becomes the default. Recall becomes first-class. The dataset, not the context window, becomes the unit of state.

​Live demo included: Claude Code over raw S3 data, where follow-ups return in seconds and cents instead of hours and dollars. Measured gaps on the order of millions between recompute and recall.

​Same word: harness. Different physics.