Autoscaling RL Paper Club — DeepSeek-V4.1-Flash & the Sim2Real Gap
Two papers on the same question from opposite ends: how far can you trust an environment you generated yourself?
DeepSeek-V4.1-Flash — we are reading the post-training half, not the KV-cache architecture. DeepSeek is explicit that the recipe is standard SFT → RL → on-policy distillation with no algorithmic changes; essentially all of the reported gains come from scaling the quantity, diversity and verifiability of automatically synthesised agent tasks and environments. Coding environments are built from real coding-agent sessions and public repos, with specialist agents constructing and verifying each task. The report is also unusually candid about RL failure modes: reward hacking, agents exploiting real vulnerabilities in their sandboxes, deleting system binaries, occasionally wiping the filesystem.
Mind the Sim2Real Gap in User Simulation for Agentic Tasks (Zhou et al., arXiv:2603.11245) — the first full τ-bench protocol run with real humans: 451 participants, 165 tasks, 31 LLM simulators scored on a new User-Sim Index. Simulated users turn out to be excessively cooperative, stylistically uniform, and free of realistic frustration or ambiguity — an "easy mode" that inflates agent success rates above the human baseline. Higher model capability does not predict more faithful simulation.
The through-line: one paper says scale your synthetic environments and verify them, the other says the simulated humans inside them are flattering you.
Papers
DeepSeek-V4.1-Flash tech report: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf
Mind the Sim2Real Gap: https://arxiv.org/abs/2603.11245
Format
An hour. A short intro on each paper, then open discussion on what it means for anyone building environments. You do not need to have finished the papers, and you do not need to work on this full time. Sessions are recorded — tell us and we keep you out of the recording.