AI Safety Weekly: Alignment from Token Zero
IMPORTANT: This session will be more technical than other sessions in the series.
Most alignment work today happens after pretraining. A model first learns from huge amounts of text, picking up all kinds of behaviors and voices along the way, and only afterwards is it trained to act as a helpful, harmless assistant. As a result, a model’s values can end up as a thin layer on top of everything it learned before, rather than something deeply rooted. But what if alignment started from the very first token of training?
In this themed session, V. Moskvoretskii (an EPFL PHD student) will present his recent work, Synthetic Persona Pretraining: Alignment from Token Zero. Instead of adding the assistant persona at the end, Synthetic Persona Pretraining (SPP) builds it in from the start: pretraining documents are annotated with first-person moral reflections based on a value constitution, and post-training then ties the assistant identity to this pretrained persona. In models of up to 3B parameters trained on 500B tokens, SPP improved how well models follow their constitution, made them more robust to jailbreaks, and reduced misaligned choices in moral dilemmas, all without hurting capabilities. Timing turns out to matter: introducing the same intervention only at the end of pretraining works noticeably worse. The technique is also part of the training of Apertus 2, the next version of the Swiss open-source LLM, so this is a chance to hear about safety work going into an open model built right here in Switzerland.
After the presentation, we’ll open the floor for discussion. Should alignment start at the pretraining stage? What are personas in LLMs, and how do they form? When in training is the best time to introduce desired properties, and can earlier really be better than later? And how do we even measure these differences, especially in small models? We can also use the session to discuss questions around open-source AI safety more broadly.
🍕 Food will be provided after the session for networking and further discussion.
📅 Tuesday, 17 November, 18:30–20:00
📍 ETH Main Building, HG D3.3.
Moderate technical background is recommended. Basic familiarity with how large language models are trained (pretraining vs. post-training/fine-tuning) is benefically. Registration is required.