Autoscaling RL Paper Club — User Models and User Simulators
A follow-up to our Sim2Real session, where a 451-person study showed LLM user simulators are an easy mode, and we split on whether to use them at all. This time both papers take the route that study left open: instead of prompting an assistant to play a user, train a model whose reward is being hard to tell apart from real users.
Persimmon (humans&, research preview) is the large version: a 550B "user model" mid-trained on public human conversation, then trained with RL against a discriminator to stay coherent over long chats. In a Multi-User Turing Test it fools an LLM judge 19.8% of the time (50% is parity) against under 1% for frontier models, but it stays coherent over 80 turns only 60.7% of the time, against 87.3% for humans.
Learning User Simulators with Turing Rewards is the controlled version: an LLM judge sees a user's history and two replies, the real one and the model's, and rates which the real user wrote; an 8B simulator is trained with RL on that score. It beats similarity and log-probability rewards on human-likeness without losing content, and in a 360-person human test it is the only model picked over the real user above chance on chat.
What we want to argue about: both make an LLM judge the new verifier, and both find it sharper than people at spotting fakes. Who checks the judge? Does either answer the Sim2Real result? And does training for human variance conflict with the tightly closed instructions we argued for last time?
Papers
Persimmon: https://persimmon.humansand.ai/blog/
Learning User Simulators with Turing Rewards (Wang et al.): https://arxiv.org/abs/2606.19336
Background — Mind the Sim2Real Gap: https://arxiv.org/abs/2603.11245
Notes: https://research.vibrantlabs.com/club/user-models-and-user-simulators
Format
An hour. A short intro on each paper, then open discussion on what it means for anyone building environments. You do not need to have finished the reading, and you do not need to work on this full time. Sessions are recorded — tell us and we keep you out of the recording.