Cover Image for Autoscaling RL Paper Club — Quality Control for RL Environments, with Kumar Anirudha
Cover Image for Autoscaling RL Paper Club — Quality Control for RL Environments, with Kumar Anirudha
Avatar for Vibrant Labs
Presented by
Vibrant Labs
Hosted By
14 Going

Autoscaling RL Paper Club — Quality Control for RL Environments, with Kumar Anirudha

Google Meet
Registration
Welcome! To join the event, please register below.
About Event

​Kumar Anirudha leads a session on quality control for RL environments, built around Sean Cai's “QC for RL Data” — a practical protocol for checking RL datasets before anyone trains on them, written from the side of the lab buying the data.

​It starts by asking what a dataset is for — training signal, a benchmark, or both — because each needs a different QA plan. It then reviews the vendor's artifacts, actually post-trains a small model on the data, and attacks the result: reward-hacking probes, stress tests of the verifier with known-good and known-bad trajectories, realism and “does the model know it's being tested” checks, and load tests of the harness. It ends by answering six questions a lab asks: is it trainable, is it real, is the grading correct, does training help, does it break anything, and is the environment sound. The floor is 19 mandatory checks, designed to be automatable.

​What we want to argue about: is reward hacking a dataset defect or a harness defect? What is the cheapest trustworthy way to measure a verifier's false positives and false negatives? Should “eval awareness” — environments full of simulation tells — be a QC gate for synthetic environments? And is a trainable-difficulty gate at intake worth it, when RL already drops tasks that are always or never solved?

​Reading

​QC for RL Data (Sean Cai): https://www.seancai.com/philosophy/good_qc_for_rl_data_v2

​Notes: https://research.vibrantlabs.com/club/quality-control-for-rl-environments

​Format

​An hour. Kumar walks through the protocol, then open discussion on what it means for anyone building environments. You do not need to have finished the reading, and you do not need to work on this full time. Sessions are recorded — tell us and we keep you out of the recording.

Avatar for Vibrant Labs
Presented by
Vibrant Labs
Hosted By
14 Going