20 Chair Lounge 001 - Multimodal Model Evaluation - Llama Beta Fellowship
20 Chair Salon is an off-the-record series hosted by Llama Beta Fellowship for the people building the agent-native era.
20 Chair Lounge 001 — Multi-Model
"Your Models Disagree With Each Other. Now What?"
Our second researcher series ran long, sparked two genuine debates that never fully resolved, and left us with one clear signal: the room wanted to go narrower and deeper next time. Everyone's stack now runs three, four, sometimes five models. Almost nobody has actually sat down to argue through what that's doing to how they route, evaluate, and pay for the work.
This session is built around that question.
What We'll Work Through
Orchestration's future. As foundation models get more natively multimodal, is the orchestration layer becoming permanent infrastructure, or something the labs eventually swallow whole? We'll put the room on record.
The trust problem. Cross-checking outputs across a multi-model stack is already happening everywhere. Is it structurally sound, or just the only thing that scales? Two sides, one debate.
The real cost. Not the API bill — the routing logic, the debugging, the invoice nobody can fully reconcile at the end of the month. We're asking everyone in the room to name a number.
When you register, tell us the most expensive or confusing multi-model decision you've made recently — we'll use these to shape the open-floor conversation live!
Format
Opening round: everyone's sharpest hot take on running multiple models
Two structured debates — the room splits into two sides, argues it out, and (usually) never fully agrees by the end
Open floor: real war stories, no topic off-limits as long as it's multimodal
No panel, and nothing leaves the room so speak freely
Agenda
1:30: Doors open
2:00: Opening round
2:30-3:00 : Debate
3:00-4:00 : Open floor + networking
Who's in the Room
Twenty people. Researchers, founders and engineers actually running multi-model stacks in production. If you're the one debugging why two models gave you contradictory answers at 2am or running each task on a separate model, this is the room for you!
Every registration is reviewed so the room stays sharp. Only those who really want to be in the room will have a spot!
A Note on RSVPs
Seats are strictly limited, so please register only if you're genuinely likely to attend. If your plans change, let us know early so we can offer your seat to someone else.
Hosted by Llama Beta Fellowship