90/30 Club (ML reading) #51: Recursive Language Models (RLMs): Scaling Beyond Context Windows
Week 51: Recursive Language Models (RLMs): Scaling Beyond Context Windows
Recursive Language Models (RLMs) reframe long-context reasoning by treating the prompt as an external environment rather than something to fully fit into the model’s context window. Instead of processing all tokens, the model interacts with the prompt through a REPL-like setup, selectively reading, filtering, and decomposing relevant parts, and recursively calling itself on subproblems. This allows RLMs to handle extremely large inputs (millions of tokens) while avoiding context degradation and maintaining strong performance. Overall, the paper shows that long-context capability is less about scaling model architecture and more about how effectively the model navigates and reasons over its input at inference time.
Join us at Mox to explore:
What exactly is the bottleneck RLM is solving memory, reasoning, or both?
Is this fundamentally different from tool use / agents, or just a cleaner abstraction?
Discussion at 20:00, (optional) quiet reading from 19:00.