

🤿 Data Deep Dive: Checking the Rubrics on Five Public Benchmarks
​Data Deep Dives are virtual sessions that walk through research at Snorkel AI with the people who did the work: a talk followed by open Q&A.
​Rubrics score model and agent outputs and supply reward signals in training. However, rubric quality itself is rarely examined. It’s usually inferred from downstream signals such as judge-human agreement or from training gains, and both mix up the rubric with the model applying it.
​RIFT (RubrIc Failure mode Taxonomy) helps practitioners detect and name failures in the rubric itself. It computes established quality signals from the rubric’s content and ties each deviation to a named failure mode. In a controlled test with defects injected into 60 expert-authored rubrics, the quality signals help detect failures with both, simple linear models and large frontier models..
​Applied to 147 tasks from five public benchmarks (domains including finance, medicine, software engineering, professional advice, and reasoning), RIFT found a large presence of missing or misaligned criterion, along with a large amount of hackable and repeated rubric items.
​This work helps researchers to understand measurement instruments at a deeper level and provides examples, guidance, and a new taxonomy to navigate and enrich rubrics.
​Join Ankit Aich, Snorkel Senior Research Scientist focused on frontier model benchmarking and failure point analysis (PhD, UIC; previously UPenn, Scale AI, FINRA). He’ll be in conversation with Jonathan Schlosser, Snorkel AI Advocate and a graduate-level instructor at UNC Chapel Hill.