

PyAI Berlin meetup with JetBrains & Pydantic, featuring GuruSup
Jodie Burchell (JetBrains) and Laís Carvalho (Pydantic) are hosting an evening PyAI Meetup at the JetBrains office in Berlin, with three talks from people building and measuring these systems, then food, drinks and time to argue about them.
The talks
Reliable, rigorous, wrong: A psychometric view of LLM evals
Jodie Burchell, Team Lead of Python Advocacy at JetBrains
Held-out test sets, calibrated LLM judges and significance tests tell you a score really changed. They do not tell you the system got better. Jodie treats LLM evaluation as a measurement problem, one psychology has worked on for over a century, and applies the psychometric idea of validity: not only "is this score reliable?" but "what evidence supports the conclusion I want to draw from it?" She starts with how benchmarks fail basic validity checks, moves to the metrics teams build for their own applications, such as "helpfulness" or "task success", and closes with four validity questions to apply to your next eval.
What one line of Code Mode does to your agent
Laís Carvalho, Developer Relations at Pydantic
If you've tried writing your own agents, you've probably tried a weather agent. My Pydantic AI weather agent has two tools, get_lat_lng and get_weather, and it does fine when I ask about two locations. So I gave it a harder question: which EU capital will be warmest this weekend? That's 27 cities and 54 tool calls, with every result going back through the model. It got there, slowly. In one of my test runs it decided 8 capitals were probably enough.
In this talk we'll add Code Mode to that agent. The model will write a short Python script that loops over all 27 cities. The script runs in Agent Sandboxes, the hosted Monty sandboxes in Pydantic Logfire, where it can only call the functions I hand it. Let's compare the runs in Logfire: tokens, cost, time, and latency using a sandbox.
From Company Knowledge to Living Skills: GuruSup's Journey with Pydantic AI and Monty
Fran Pastor Agustí, CTO at GuruSup
Fran walks through an agent that investigates a customer’s order sync incident across Gmail, MongoDB and Logfire, then records it in Notion. Both agents are built with Pydantic AI and return typed Pydantic results. The talk follows how the skill evolves. It starts as a Markdown procedure that guides the tool calls one by one. Then Code Mode lets the model write Python that chains the calls, running in the Pydantic Monty sandbox. A second agent, the improver, writes one program for the whole investigation, which is saved in the skill so later runs just execute it. When it fails, the agent falls back to the Markdown procedure and still finishes the task. The improver then repairs the program from the Logfire trace, and a person approves the fix as a new revision of the skill. He closes with how GuruSup does this behind the scenes, turning company knowledge into executable skills without users having to build and refine the workflow themselves.
Schedule
18:00 Doors open
18:30 Talks and Q&A
19:30 Food, drinks and networking
21:00 Close
Who should come
Python developers, ML engineers and anyone building with LLMs who wants to hear what happens after the demo. Spots are limited to 100.