Agent Simulation: Stress-Test Agents Against Synthetic Users
A live walkthrough of automated agent simulation — running your AI agent through hundreds of realistic, multi-turn conversations against synthetic users before a single real user touches it.
Agents fail in ways static, single-turn evals miss: they lose the thread over a long conversation, cave to a manipulative user, hallucinate under pressure, or never reach the goal. Hand-writing these test conversations doesn’t scale, and production is an expensive place to find out.
In this 45-minute session with a live coding demo, we’ll run eq sim against an Orq agent end to end. From a one-line agent description, the framework generates personas (varying patience, assertiveness, technical level, plus emotional arcs like escalating, hostile, manipulative) and scenarios (each with a goal, pass/fail criteria, and adversarial strategies like contradictory, evasive, or multi-intent). It then drives every persona × scenario conversation with a user-simulator agent, while a judge agent scores each turn and decides when the goal is met, or broken.
You’ll see how to read the results: goal-completion rates, criteria met/violated, hallucination risk, tone, and turn efficiency, all uploaded to Orq as an experiment with full traces, so a regression becomes a gate you can run on every agent change.
What we’ll cover:
Why multi-turn simulation catches failures single-turn evals don’t
Live demo: eq sim run from agent description → generated personas + scenarios → scored conversations
How the user-simulator and judge agents drive and grade an adversarial multi-turn conversation
Reading goal-completion, criteria, and quality scores; freezing inputs for apples-to-apples comparison across agent versions
Wiring simulation into CI as a pass/fail gate
Who it’s for: AI engineers and platform teams shipping agents to production. Familiarity with LLM agents assumed; no prior testing-framework experience needed. 45-minute session with live coding demo, followed by Q&A.
Host: Bauke Brenninkmeijer, AI Research Engineer — Orq.ai research team