LLM Paper Club - Everything Distillation Survey Talk
bweissmann is giving us an overview of many things distillations!
Survey paper style covering the following with some nice high level overviews to catch up on quickly!
So many great papers below!
https://arxiv.org/pdf/2601.20802 < Hübotter et al Reinforcement Learning via Self-Distillation https://siyan-zhao.github.io/assets/img/opsd/opsd_v3.pdf < Zhao et al Self-Distilled Reasoner https://arxiv.org/pdf/2607.05184v1 < Kaur et al Rethinking On-Policy Self-Distillation for Thinking Models https://www.appliedcompute.com/research/relevance-masked-self-distillation < Applied Compute RMSD https://www.appliedcompute.com/platform/productionizing-self-distillation-methods < Applied Compute Pt 2 https://arxiv.org/pdf/2606.03532v1 Guo et al When Should the Teacher Move? Temporal Coupling and Stability in Self On-Policy Distillation https://www.youtube.com/watch?v=ZTA0GwpAUak < Applied Compute AIEngineer talk And for background, @tedk's paperclub on OPSD in February was great (https://www.youtube.com/watch?v=CrJp0sd6IGI) theres a nice interview with the authors of the Hübotter paper here (https://www.youtube.com/watch?v=OgEGV7apEzI) and Dwarkesh's mini Sasha Rush explainer is always fun (https://www.youtube.com/watch?v=wxOZWD6wYVY)
---
we need YOU to volunteer to do rapid-fire recaps and explanations of our remaining papers on the board: https://app.sli.do/event/bNV6mo3BFGhe8Bqzb1tonb/live/questions
please sign up in #llm-paper-club in https://www.latent.space/p/community discord
recordings at LatentSpaceTV