Learning Layer Paper Reading Club - Week 36 - Reward Hacking Probes
This week's paper: Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
Link: https://arxiv.org/abs/2609.19101
Abstract:
When LLM agents are evaluated on coding tasks, they sometimes "reward hack": they game the test harness instead of solving the problem. Catching this usually means running another LLM as a monitor, which is slow and costly, and it can miss hacks the model never talks about.
This paper looks at what reward hacking looks like inside the model. Working with frontier open-source models (Kimi K3, GLM 5.2, Qwen 3.8 Max), the authors show that simple difference-of-means vectors in activation space capture reward hacking coherently, generalize across different hacking behaviors, and detect hacks reliably. GLM 5.2, for example, hacks in 57.2% of DeepSWE rollouts and 73% on SWE-bench.
These probes are nearly free to run and, at a matched false positive rate, perform comparably to LLM monitors. They also work on the chain-of-thought to predict hacks in later actions, which makes online detection possible, and they surface some undesirable behaviors that LLM monitors miss.
Discussion Topics:
Probes beat the LLM monitor on Kimi K3 but lose on GLM 5.2. What explains the gap, and which should you trust?
Is a difference-of-means direction really "reward hacking", or a proxy for something nearby like deception or task giving-up?
If you train against an internal-representation monitor, how quickly does the model learn to hide the signal?
Predicting hacks from the chain-of-thought enables online intervention. What should happen when the probe fires?
The probes transfer to non-SWE evaluations. How far should we expect that to go?
What would make evaluation numbers trustworthy when 57-73% of rollouts may be hacked?
Anyone interested in AI evaluation, interpretability, or agent safety will find something here. No preparation required — come read and discuss with us.
What is the Learning Layer Labs Paper Reading Club?
An initiative from https://www.learninglayer.ai, a lab with the goal of reducing AI anxiety in the world.
What is the format?
Discussion based. Expect a low pressure environment to share insights and opinions with the group.
What are the group goals?
Stay on top of AI research and improve understanding of AI fundamentals + math.
Who is welcome?
Everyone! Try to put in at least some time on the paper and come prepared with questions or things you'd like to discuss, but it's ok to just show up!
Learning Layer Labs team:
Thomas Redfern ( The backbone who runs every paper reading )
Mat Allen (The new fella bring order and organization)
Devinder Sodhi (The guy you talk to for sponsoring)