90/30 Club (ML reading) #49: Mamba-3: Rethinking Sequence Models Beyond Transformers
Week 49: Mamba-3: Rethinking Sequence Models Beyond Transformers
Transformers aren’t bottlenecked by capability anymore, they’re bottlenecked by inference cost. As models scale and agentic workflows explode, the real constraint becomes how efficiently we can run these systems, not just train them.
This paper introduces Mamba-3, a new state space model architecture that rethinks sequence modeling from an inference-first perspective. Instead of relying on attention and KV caches, Mamba-3 builds on structured recurrence to achieve linear compute, constant memory, and improved hardware efficiency, without sacrificing model quality.
The result is a model that not only competes with Transformers, but begins to shift the performance efficiency frontier, especially in long-context and deployment-heavy settings.
Join us at Mox to explore:
How exponential trapezoidal discretization effectively introduces a learned convolution inside the recurrence, and why this matters for expressivity
Why complex-valued state updates (via a RoPE-like mechanism) unlock true state tracking, fixing a core weakness of linear models
Discussion at 20:00, (optional) quiet reading from 19:00.