Vals AI: Measuring Social Mobility
Vals AI just released three new benchmarks, Financial Health Bench, Access to Justice Bench, and Immigration Bench, to measure how helpful LLMs are at answering some of the most pressing questions people face.
As AI becomes part of how people make major decisions, we need to know whether it works for everyone, not only the people who already have access to good advice. These benchmarks are our attempt to find out.
We’ll open with a happy hour, then move into short presentations on each benchmark, a fireside chat, and audience questions.
Financial Health Bench: Advised by the Financial Health Network, this benchmark sees AI models coach a simulated client through everyday money problems like debt, rent or student loans. Their advice is graded against a checklist written by financial practitioners.
Access to Justice Bench: Advised by Stanford's Legal Design Lab, we evaluate the extent to which AI models can act as a reliable first point of contact for clients with a legal problem, the kind of role an intake worker or a court self-help center would play. Understanding the person's situation, giving accurate legal information, and steering the individual towards local help all are assessed.
Immigration Bench: Built with the Talent Mobility Fund, it tests AI models on hundreds of U.S. immigration questions written by experts, from student visas to DACA. Each answer is graded against an expert-written rubric.
Agenda
6:00pm: Doors open, snacks and refreshments
6:30pm: Benchmark presentations
7:00pm: Fireside chat
7:30pm: Audience Q&A
8:00pm: Open mingling + discussion