EPFL x MATS Research Talk
Join us for a research talk by three MATS fellows (Google DeepMind Stream) Euodia Dodd, Hardik Bhatnagar, and Lennart Finke, who have recently completed a research project mentored by Mary Phuong.
This talk is jointly organized by the EPFL AI Center room ELE117, the ML4D Lab, and the MATS Research Program as part of the AI Fundamentals Seminar Series.
Title
Towards Robust Model Organisms for Eliciting Latent Knowledge
Abstract
A key problem in AI safety is eliciting latent knowledge (ELK), extracting information from a model that is misaligned and more knowledgeable than its overseer. For example, a misaligned model used to scan code for vulnerabilities may want to withhold its findings, i.e., sandbag, in order to exploit them later.
To study ELK empirically, we need model organisms that mirror this setting. Existing model organisms fall short in several respects. In this work, we train model organisms to withhold solutions in a vulnerability detection task.
We combine transcript distillation with counter-prompting and tamper-resistance training to instill three properties that we argue are critical for a useful ELK testbed:
Selectivity: The model sandbags when it thinks the overseer doesn't know the answer and complies otherwise.
Robustness: The model resists elicitation.
Retention: The model retains a knowledge advantage over the overseer, as opposed to losing the relevant capability.
We train a sequence of model organisms and show that our strongest model substantially outperforms prior models on robustness while keeping the other two properties intact.
We hope that our model organisms provide a useful ELK testbed for the community, and that our methodology sparks the development of increasingly challenging model organisms.
More details here https://memento.epfl.ch/event/ai-center-seminar-ai-fundamentals-series-mats-fell/