Exploration Hacking: Can LLMs learn to resist RL training?
Summary
Reinforcement learning (RL) has become central to post-training language models for reasoning, agentic capabilities, and alignment. Unlike supervised learning, on-policy RL depends on models exploring different actions and thereby generating the data from which they learn.
This creates a potential failure mode: a sufficiently capable model might strategically avoid high-reward behavior, preventing RL from reinforcing it and influencing the outcome of its own training. We call this exploration hacking.
In this talk, Joschka Braun will present recent ICML 2026 research studying exploration hacking empirically. The researchers construct model organisms deliberately trained to under-explore and test whether they can resist RL-based capability elicitation on biosecurity and AI R&D tasks. They then evaluate potential countermeasures and audit frontier models for the reasoning capabilities and behavioral propensity required for exploration hacking.
The findings are nuanced: the model organisms can resist RL-based elicitation, but current examples are detectable using monitoring and weight noising. Meanwhile, frontier models in the study can reason about exploration hacking when given sufficient training context, but do not show a natural propensity to act on it.
📅 Wednesday, 22 July 2026, 18:00 - 19:00 CEST
What we will cover
- Why successful RL post-training depends on sufficient exploration.
- What exploration hacking is and why it could undermine capability elicitation and alignment training.
- How model organisms of exploration hacking can be constructed and evaluated.
- When these models successfully resist RL-based capability elicitation.
- What monitoring, weight noising, and supervised fine-tuning reveal about possible countermeasures.
- Whether current frontier models are capable or inclined to exploration hack.
- Open questions about when exploration hacking could emerge and how RL post-training can be made more robust.
Format
30-minute presentation followed by Q&A via Slido.
Who Should Attend?
This talk is intended for technical AI safety researchers, ML and computer science students, and anyone interested in how language models interact with their own training processes. Basic familiarity with machine learning will be helpful, but no prior expertise in reinforcement learning is required.
Speaker Bio
Joschka Braun is an AI safety researcher studying the training dynamics and potential misaligned behavior of advanced language models. His recent work at MATS, mentored by David Lindner, Roland Zimmermann, and Scott Emmons, investigates whether models can influence reinforcement-learning outcomes by strategically changing their exploration behavior. This work was accepted at ICML 2026.
Previously, Joschka researched the reliability of steering vectors at the Krueger AI Safety Lab and representation-engineering applications at the Health-NLP group. He holds a master’s degree in Machine Learning and a bachelor’s degree in Computer Science from the University of Tübingen.
This talk presents work co-led by Joschka Braun, Eyon Jang, and Damon Falck as equal contributors and conducted through MATS. The complete author list is available in the paper.
Paper: https://arxiv.org/abs/2604.28182
ICML page: https://icml.cc/virtual/2026/poster/64674
Website: https://joschkacbraun.github.io/
LinkedIn: https://www.linkedin.com/in/joschka-braun/
X: https://x.com/BraunJoschka
Original paper thread: https://x.com/BraunJoschka/status/2051324364943921368
And share with others who might also be interested! https://lnkd.in/p/e-rh_xNx