From Signal to PR: Evaluating and Shipping AI Agents That Work
When an agent fails in production, finding the bad output is only the beginning. Teams still have to identify the pattern, trace it back to the underlying cause, propose a fix, and verify that the change improved the full agent workflow.
This session shows how Arize AX helps teams run that loop as a system.
We’ll start with Signal, the always-on worker that reviews production traces and surfaces emerging failure patterns. From there, we’ll show how managed agents can investigate those issues with repository context, propose fixes, and carry the workflow through to a pull request. Then we’ll use Agent-as-a-Judge and full-agent experimentation to verify that the change improved the agent’s behavior, not just a single output.
What we'll cover (30 min presentation + 15 min Q&A)
Signal— how an always-on worker continuously reviews production traces, clusters failures into issues, and writes up root cause and suggested next steps
Managed agents — how to use Agent Studio to start from a guided preset or build a fully custom worker that investigates issues, works with repo context, and propose fixes as PRs
Agent-as-a-Judge — building evaluators from plain-language criteria inside a Claude Code sandbox, without column mapping or brittle rubrics
Full-agent experimentation — running experiments against a live endpoint to confirm a change improved across the entire harness
Who this is for
AI engineers and agent builders shipping agents into production, along with PMs and technical leads responsible for agent quality and reliability.
What you'll leave with
A clear mental model for the detect → investigate → verify → improve workflow, and a repeatable way to run it against a production agent. We'll show the full loop live; follow-up resources will point you to code and docs to run it yourself.
Format
30-minute presentation, 15-minute live Q&A
Level
Intermediate. Familiarity with LLM agents, traces, or evals helps, but we won't assume you've already solved production reliability.