

Research Paper discussion: Model Distillation in the Era of LLM:s
Seminar and Discussion on Model Distillation in the Era of Large Language Models!
Distillation has made headlines as a way for one AI model to learn from another, with or without the consent of the model provider. Beyond the headlines, it is a fundamental concept that every ML engineer or researcher should be familiar with. It is an essential part of the pursuit of efficient neural networks and can therefore be found in many places where models are deployed to production.
The event is open to both relative beginners and those who want to explore distillation in more depth. In the first half, we'll cover Hinton et al.'s classic paper on knowledge distillation and look at what a smaller model can learn from a teacher’s predictions. In the second half, we'll discuss the more advanced concept of generalized knowledge distillation and the paper On-Policy Distillation of Language Models, also coming out of Google.
Please check out the papers beforehand. Both of them are a good and concise read. However, you can also check out these articles if you do not find them approachable: https://www.ibm.com/think/topics/knowledge-distillation (classic distillation) https://thinkingmachines.ai/blog/on-policy-distillation/ (distillation of language models)
Free to attend. Spots are limited, so sign up to make sure you get a seat. Room 101162!
Organised by UU AI Society.