Teacher Reward Hacking in Language Model Distillation
Post-training of language models often involves two steps:
distillation, where a student imitates a larger teacher model;
RLHF, where the model is aligned using a reward model.
While RLHF can suffer from reward hacking (overfitting to the reward model), we ask whether a similar issue, teacher hacking, can occur during distillation, due to the teacher being an imperfect proxy for the true distribution.
Using a setup with an oracle, teacher, and student model, we show that teacher hacking emerges with fixed offline data and can be detected via deviations from expected convergence patterns.
In contrast, online data generation, especially with high data diversity prevents hacking. This sheds light on the tradeoffs in using distillation to train robust LMs.