Cover Image for Scaling Visual Intelligence for High-Resolution and Long-Form Video
Cover Image for Scaling Visual Intelligence for High-Resolution and Long-Form Video
Led by Mayank Bhaskar and Benedict Emoe-Kabu. Part of the Cohere Labs Open Science initiative https://cohere.com/research/open-science
Hosted By

Scaling Visual Intelligence for High-Resolution and Long-Form Video

Registration
Past Event
Welcome! To join the event, please register below.
About Event

​What would it take for a robot or a wearable assistant to watch an entire day and help a blind person find their keys? Current vision systems make this prohibitively expensive because they split every image into a fixed grid of patches. As a result, doubling the resolution quadruples the token count, even when the added pixels carry no new information (e.g., a blank wall), and every video frame adds hundreds more tokens, even when it barely differs from the last. This talk presents region-based representations that encode each meaningful entity as a single token that persists across frames, so computation scales with what the scene contains rather than with resolution or video length. I will show how to generate region tokens efficiently, extend them to video, and align them with language. With region tokens, an hour of video takes fewer tokens than a patch-based encoder needs for a twenty-second clip, and performance improves on tasks ranging from segmentation and retrieval to episodic memory localization and spatial reasoning.

Savya Khosla is a Ph.D. student in Computer Science at the University of Illinois Urbana-Champaign, advised by Derek Hoiem and Alexander Schwing. His research focuses on efficient visual representations, long-video understanding, and multimodal learning. He develops region-based representations that enable vision models to perceive more detail and remember longer videos at a fraction of the compute cost. His work has resulted in multiple publications at CVPR, NeurIPS, ECCV, and ACL. Before his Ph.D., he was a software engineer at Google, working on Search, and has since conducted research at Meta, Adobe, Mila, NUS, and Ai2.

Location
https://meet.google.com/pqd-higu-fwa
Led by Mayank Bhaskar and Benedict Emoe-Kabu. Part of the Cohere Labs Open Science initiative https://cohere.com/research/open-science
Hosted By