Cover Image for Research Paper discussion: Model Distillation in the Era of LLM:s
Cover Image for Research Paper discussion: Model Distillation in the Era of LLM:s
Hosted By
Private Event

Research Paper discussion: Model Distillation in the Era of LLM:s

Hosted by UU AI Society
Registration
Approval Required
Your registration is subject to host approval.
Welcome! To join the event, please register below.
About Event

​Seminar and Discussion on Model Distillation in the Era of Large Language Models!

​Distillation has made headlines as a way for one AI model to learn from another, with or without the consent of the model provider. Beyond the headlines, it is a fundamental concept that every ML engineer or researcher should be familiar with. It is an essential part of the pursuit of efficient neural networks and can therefore be found in many places where models are deployed to production.

​The event is open to both relative beginners and those who want to explore distillation in more depth. In the first half, we'll cover Hinton et al.'s classic paper on knowledge distillation and look at what a smaller model can learn from a teacher’s predictions. In the second half, we'll discuss the more advanced concept of generalized knowledge distillation and the paper On-Policy Distillation of Language Models, also coming out of Google.

​Please check out the papers beforehand. Both of them are a good and concise read. However, you can also check out these articles if you do not find them approachable: https://www.ibm.com/think/topics/knowledge-distillation (classic distillation) https://thinkingmachines.ai/blog/on-policy-distillation/ (distillation of language models)

​Free to attend. Spots are limited, so sign up to make sure you get a seat. Room 101162!

​Organised by UU AI Society.

Location
Ångströmlaboratoriet
Lägerhyddsvägen 1, 752 37 Uppsala, Sweden
Hosted By