🤿 Data Deep Dive: Checking the Rubrics on Five Public Benchmarks

​Data Deep Dives are virtual sessions that walk through research at Snorkel AI with the people who did the work: a talk followed by open Q&A.

​Rubrics now score model and agent outputs and supply reward signals in training. Rubric quality, though, is rarely examined directly. It’s usually inferred from judge–human agreement or from training gains, and both mix up the rubric with the model applying it.

​RIFT (RubrIc Failure mode Taxonomy) evaluates the rubric itself. It computes quality signals from the rubric’s content and ties each deviation to a named failure mode. In a controlled test with defects injected into 60 expert rubrics, the signals identified the injected mode with 75.0% accuracy. An LLM judge given the same rubrics reached 56.7%.

​Applied to 147 tasks from five public benchmarks (finance, medicine, software engineering, professional advice, and reasoning), RIFT found that the mechanics were mostly sound. Bundled requirements appeared in 7% of tasks and subjective wording in 10%. The validity problems were far more common:

• ​Missing criteria: 50% of tasks (90% in medicine)

• ​Misaligned criteria: 48%

• ​Ungrounded verification: 37%

• ​Redundant criteria: 35%

​Join Ankit Aich, Snorkel Senior Research Scientist focused on frontier model benchmarking and failure point analysis (PhD, UIC; previously NIH, UPenn, Scale AI, FINRA). He’ll be in conversation with Jonathan Schlosser, Snorkel AI Advocate and a graduate-level instructor at UNC Chapel Hill.