LLM Paper Club - Everything Distillation Survey Talk
bweissmann is giving us an overview of many things distillations!
Survey paper style covering the following with some nice high level overviews to catch up on quickly!
So many great papers below!
https://arxiv.org/pdf/2601.20802 < Hübotter et al Reinforcement Learning via Self-Distillation https://siyan-zhao.github.io/assets/img/opsd/opsd_v3.pdf < Zhao et al Self-Distilled Reasoner https://arxiv.org/pdf/2607.05184v1 < Kaur et al Rethinking On-Policy Self-Distillation for Thinking Models https://www.appliedcompute.com/research/relevance-masked-self-distillation < Applied Compute RMSD https://www.appliedcompute.com/platform/productionizing-self-distillation-methods < Applied Compute Pt 2 https://arxiv.org/pdf/2606.03532v1 Guo et al When Should the Teacher Move? Temporal Coupling and Stability in Self On-Policy Distillation Watch on YouTube < Applied Compute AIEngineer talk And for background, @tedk's paperclub on OPSD in February was great (Watch on YouTube) theres a nice interview with the authors of the Hübotter paper here (Watch on YouTube) and Dwarkesh's mini Sasha Rush explainer is always fun (Watch on YouTube)
---
we need YOU to volunteer to do rapid-fire recaps and explanations of our remaining papers on the board: https://app.sli.do/event/bNV6mo3BFGhe8Bqzb1tonb/live/questions
please sign up in #llm-paper-club in https://www.latent.space/p/community discord
recordings at LatentSpaceTV