Where it broke, and why: the production stack for agentic development
A deploy goes out and latency climbs. Service-level telemetry tells you which service and which request path. It usually can't tell you which function changed behavior, or which commit did it. Now that agents ship more changes a day, that gap is where most of the investigation time goes.
This session follows one regression from the service down to the function. We'll start with OpenTelemetry traces and metrics stored in ClickHouse to isolate the service, deployment and request path. Then we'll compare per-function runtime data against each function's own baseline to find what changed and what it did when it failed. Trace IDs join the two datasets. Last, we'll hand the result to an engineer or a coding agent over MCP.
Agenda
Why more changes per day makes investigation the bottleneck.
What service-level telemetry answers, and where it runs out.
Joining traces and function-level data on trace IDs.
Live walkthrough: one change through PR, deploy, regression and fix.
Takeaways and 15 minutes of Q&A.
Takeaways
When OpenTelemetry data is enough, and when you need function-level data.
How to join the two on trace IDs so one query path runs from service to function.
How to give an engineer or agent the failing function's inputs and behavior before they start a fix.
A staged path to automatic rollback: recommendation first, policy in your CD later.
Speakers: Dale McDiarmid from ClickHouse and May Walter, Co-founder and CTO of Hud.
45 minutes including Q&A. Register to get the recording if you can't attend live.