RocketRide Office Hours: LLM Judge or Jev?
Last month someone beat the Ender Dragon in Minecraft with Jev, a model that can’t write a single word. Jev, from TypeSafe, only makes decisions: you ask it a typed question, and it gives back an answer and how sure it is. Vercel already moved its “is this command safe to run?” check for agents onto it.
Part 4 of our fortnightly agentic building series: an hour on whether the thing grading your agent should be an LLM that explains itself, or a decision model that just says yes, no, or “I’m not sure.”
What we’ll cover
Why “I’m not sure” might be the most useful thing a judge can say, and how that brings back from Session 3
An email that tries to talk the judge into saying yes, and why the fix was a sharper question, not a smarter model
Where it falls over: an independent test caught it answering “Yes” to every question on one legal task
What moving Rocket Ralph’s grading loop onto Jev could look like on RocketRide
Agenda: about 10 minutes on the topic, then open discussion. Bring your evals, your judges and your disagreements.
Runs in the RocketRide Discord. Voice or chat, both count. Recorded, with a written summary after. Part 4 of Agentic Building. You don’t need to have been at the earlier parts.
Join here: Join Discord Server