Skill Issues: How We Evaluate Data Agents For Performance and Costs

​Evals are hard, but data engineering evals are harder. Most benchmarks stop at text-to-SQL because read-only tasks are the easiest to test. Data engineering agents don't just produce queries. They ingest files, modify pipelines, replace tables and alter the state of the lakehouse in innumerable ways, often asynchronously and autonomously. An agentic lakehouse is the necessary foundation, but what about the model and the harness that consume tokens on top? How do we know which LLM to trust, how much it costs, how long it takes?

​In this session, Jacopo Tagliabue will talk about the methodology behind "Skill Issues", our open-source framework for evaluating and optimizing AI agents on real data engineering tasks. As you can only optimize what you measure, the journey starts with building real-world evals. He will cover how we generate realistic workloads from production traces and synthetic personas, how we go from a user's intent to the DAG code that expresses it to the lakehouse operations that DAG produces, and how that chain compiles into deterministic SDK checks that audit what actually changed rather than grading a chat log.

​Finally, he will cover how these evals become the foundation for optimizing agent skills with GEPA, treating skills and CLAUDE.md as hyperparameters of a compound AI system. Through this webinar you will learn the importance of programmable, Git-able data infrastructure, best practices on data engineering evals, and optimization strategies for token consumption.