

Moritz Wagner - KV-Surgeon: Compensation-Aware KV-Cache Compaction
Repeatedly querying a long document or codebase requires storing and attending to a large KV cache. Offline cache compaction offers a way to pay a one-time compression cost and reuse a much smaller cache across subsequent requests. But which entries should we retain? An entry's importance depends on what remains and on how well the remaining entries can compensate for its removal.
This talk will present KV-Surgeon, a method that brings compensation into cache selection itself. Starting from attention-output reconstruction, we derive a quadratic surrogate that admits closed-form deletion scores and compensating updates. This makes it possible to select entries according to the error remaining after compensation, rather than their individual contributions. We will explain the approximation, its connection to Optimal Brain Surgeon, and the empirical evidence for compensation-aware selection. Finally, we will examine the path toward practical serving: uniform head budgets, absorbing attention multipliers into keys, and measuring the resulting trade-offs between accuracy, memory, and inference throughput.
Moritz Wagner is a doctoral researcher at Zuse Institute Berlin working on efficient deep learning. His research focuses on compressing large language models and their inference state, particularly through pruning and KV-cache compaction. He studies how adapting the remaining parameters or cache entries can compensate for what compression removes, combining optimization-based methods with empirical evaluation.