Inside Kimi K3: Delta Attention, Gated MLA and 1M-Token Reasoning and Maths

​Moonshot AI’s Kimi K3 is not simply another large Transformer. It combines 2.8 trillion total parameters, 104 billion active parameters per token, 896 routed experts and a one-million-token context window in a distinctive hybrid architecture.

​But how does the architecture actually work?

​In this session, we will follow a token from its initial embedding through the full Kimi K3 backbone and explain the mathematics using intuitive examples.

​We will explore:

• ​how Kimi Delta Attention maintains a fixed-size, editable working memory;

• ​why the model uses three KDA layers followed by one Gated MLA layer;

• ​how MLA periodically performs global softmax attention over a compressed historical archive;

• ​how Attention Residuals allow later layers to revisit useful representations from earlier depths;

• ​how Stable LatentMoE sends compact representations to a small group of selected specialists;

• ​and how these components work together to support long-context reasoning without making every layer pay the full cost of global attention.

​This will be a technical session, but the aim is not to recite equations. The goal is to build a clear mental model of why the architecture was designed this way.

​Who should attend?

​This session is designed for:

• ​AI and machine-learning engineers;

• ​quantitative researchers and data scientists;

• ​LLM and RAG architects;

• ​researchers interested in linear attention, MoE and long-context systems;

• ​technically curious professionals who want to understand modern model architecture beyond benchmark scores.

​You do not need to have read the Kimi K3 technical report. A basic understanding of Transformers and attention will be helpful.