Benchmarking AI agents & tool use with Harbor and Arize Phoenix
When you change an agent's model, prompt, tools, or environment, how do you know whether the new version performs better? Production traces show what happens in real applications. A controlled benchmark lets you compare configurations against the same tasks before you ship.
In this demo session, we will walk through an end-to-end agent benchmark with Harbor, then analyze the results in Phoenix. Harbor is an open source framework that runs agents against tasks in sandboxed environments and uses verifiers to score the resulting state. The Phoenix plugin records those runs as versioned datasets and experiments, with scores, errors, and traces for each run. This gives teams a repeatable way to compare agents and models, distinguish behavioral failures from infrastructure problems, and inspect the trajectory behind an individual result.
During the demo, we will use the benchmark suite Arize built to test Phoenix. Each task reproduces a real Phoenix workflow, such as investigating project traces or annotating failures. Harbor starts a sandboxed environment with a Phoenix server and seeded data, runs the agent, and evaluates the final state.
We will run the same tasks against PXI, Phoenix's built-in agent, and coding agents such as Claude Code and Codex using the Phoenix MCP server, the px CLI, or Phoenix skills. You will see how the results help compare product interfaces, diagnose failures, and decide what to improve.
We will also explain the design choices behind the benchmark, how Harbor runs and scores each task, and how to keep the benchmark useful as your agents, models, and tools change.
Covered in this session
How to structure a Harbor benchmark with task instructions, environments, agents, and verifiers
How to turn a real agent workflow into a repeatable test with a measurable final state
How to run the same task set across different agent, model, and tool configurations
How to compare scores, infrastructure failures, and traces in Phoenix so you can diagnose regressions and prioritize improvements
How Harbor launches a sandbox with a Phoenix server and seeded data
How the same suite can run against PXI or coding agents with the Phoenix MCP server, px CLI, and skills
Who this is for
This session is for software engineers who build agents or the tools agents use, including MCP servers, CLIs, and skills. It is also for product managers responsible for agent quality, developer experience, or release decisions who need a consistent way to compare behavior across product interfaces and configurations.
What you will leave with
A repeatable way to compare agents and models on the same tasks, distinguish behavioral failures from infrastructure problems, and inspect the trajectory behind an individual result.
Format
30 min content + 15 min Q&A
Level
Intermediate. You should understand the basics of agents and be comfortable reading traces. Familiarity with evals or sandboxed environments will help, but no prior Harbor experience is required.