Evaluation of AI Agents: Quality vs Cost
We can use various agents: Claude Code, Codex, Aider with DeepSeek, pi.dev, Mirror with Beam and others, but do we choose them based on their real capabilities or marketing? Let's check in practice: give them the same tasks and see what happens.
We will perform tasks from simple to tricky. Each agent works alone, then we compare results.
What we look at
Whether the solution works and handles edge cases
How fast and efficient the generated code was
Token usage and their cost
How many times a human had to step in and fix the direction
The same model in a different tool can perform differently. We want to see how the harness changes the model's results on the same task.
Evening plan
Quick intros, who starts with what.
Rounds from simple to tricky, with various challenges.
Results board.
Discussion, why one succeeded and another didn't; what stacks, prompts, configurations we have.
Technical requirements: Linux or macOS.
It's good if you have your own tokens for the agent; if not, please inform before the event.
For whom: for everyone who codes with AI and wants to see how different setups perform on real tasks.
Have an idea for a tricky task? We'll be able to run and test it if we have time.