Paper Club: Preference Learning with Lie Detectors can Induce Honesty or Evasion

​Technical Note: This event is intended for participants with a technical background. We strongly encourage reading the paper ahead of time to fully engage with the discussion.

​Last session, we discussed how Chain-of-Thought (CoT) reasoning is transparent when models need to "think out loud" to arrive at the answer. This week, we investigate what happens when you detect lies by looking directly at a model's internal states, and when you try to use these "lie detectors" to train models to be more honest.

​The core tension: we want our AI systems to be honest, but can prefer lies we are convinced by over uncomfortable truths. Will including a lie detector in the training process make models more honest? Or will they simply train models to hide their lies better?

​The results reveal a delicate balance. The outcome hinges on three key factors: the detector's accuracy, the training algorithm, and the regularization strength. A high true-positive rate (TPR > 90%) pushes models toward honesty, while a less effective detector incentivizes deception. Most strikingly, on-policy algorithms like GRPO, which allow the model to explore and generate responses, learn to be deceptive far more effectively than off-policy methods like DPO. The fine-tuning process can turn a model from 95% deceptive to 95% honest based on these specific choices. The findings suggest that while detector-in-the-loop training is not a silver bullet, it can be a powerful tool for alignment if implemented with care. Join us to discuss this means for building honest AI systems.

​The paper can be found here: https://arxiv.org/abs/2505.13787