Rulin Shao - Context Language Models

​Bio:
Rulin Shao is a PhD student at the University of Washington, advised by Prof. Pang Wei Koh and Prof. Luke Zettlemoyer. She was also a visiting researcher at Meta, working with Scott Yih and Mike Lewis. Her research focuses on building AI systems with progressively greater agency, enabling them to actively gather information, learn from experience, and efficiently manage information across model weights, context, and storage to solve complex tasks.

Description: In this talk, I will present our recent work on Context Language Models (CLMs), in which we enable LMs to natively manage their own context. We implement CLMs by treating context as a file and allowing the model to make unrestricted updates to it. This allows the model to learn what information is most important to retain in context and naturally extends to multi-agent systems where multiple agent contexts coexist as files. CLMs built zero-shot with existing models outperform state-of-the-art context-management strategies across a variety of tasks: BrowseComp-Plus, TerminalBench2, EdgeBench, and a 24-hour multi-repository agent-swarm task. Moreover, by shifting context management from external harness control to intrinsic model behavior, CLMs naturally enable both in-context and parametric learning of context-management strategies. We show that CLMs can be steered and improved through in-context skill evolution and online reinforcement learning. Finally, we co-design Suffix Cache Reuse for CLM serving, further reducing server-side compute by 35% relative to standard SGLang at matched performance. At the end of the talk, I will also share related future directions I’m excited about and would love to exchange thoughts on, including CLMs for agent swarms, cache optimization for CLMs, and CLM scaling.