The KV Cache Bottleneck: Architectural Alternatives
In autoregressive decoding, each new token recomputes the keys and values for all preceding tokens. This cache grows linearly with context length, and in long-context serving it becomes the dominant memory and bandwidth cost - quite often more limiting than raw compute itself.
In this session, Abhay Kumar, co-founder of FrontiersMind, shall walk through the main architectural responses to this problem and what each one costs us. Grouped-query and multi-query attention bring down the number of KV heads. Cross-layer sharing Cross-Layer Attention and the shared KV cache in the recent Gemma models, wherein the later layers reuse key-value states from the earlier ones, bring down the number of cache-producing layers. Sliding-window attention: DeepSeek's multi-head latent attention compresses keys and values into a joint latent.
Thereafter, the talk turns to the one assumption that none of these questions: that both a key and a value have to be written at every single step. The session shall cover Grouped Value Attention, FrontiersMind's recent work on caching only the values and reconstructing keys on demand through a learned map which folds into the query at inference - why the naive version of this idea fails, what the initialisation scale between queries and reconstructed keys has got to do with whether training recovers at all, why RoPE breaks the trick, and what the arithmetic finally buys us.
About Speaker:
Abhay Kumar is co-founder of FrontiersMind, an AI research lab focused on efficient small and medium-sized language models optimised for enterprise and real-world deployment. He has previously worked on the Falcon pre-training team at TII in Abu Dhabi and is an author of ZClip, an adaptive gradient-spike mitigation method for LLM pre-training.
Pre-read
Join online: meet.google.com/iwy-hvkv-mfs
Looking forward to seeing you!