Cover Image for PyAI Berlin meetup with JetBrains & Pydantic, featuring GuruSup
Cover Image for PyAI Berlin meetup with JetBrains & Pydantic, featuring GuruSup
Avatar for PyAI
Presented by
PyAI

PyAI Berlin meetup with JetBrains & Pydantic, featuring GuruSup

Register to See Address
Berlin, Germany
Registration
Welcome! To join the event, please register below.
About Event

​Jodie Burchell (JetBrains) and Laís Carvalho (Pydantic) are hosting an evening PyAI Meetup at the JetBrains office in Berlin, with three talks from people building and measuring these systems, then food, drinks and time to argue about them.

​The talks

​Reliable, rigorous, wrong: A psychometric view of LLM evals

​Jodie Burchell, Team Lead of Python Advocacy at JetBrains

​Held-out test sets, calibrated LLM judges and significance tests tell you a score really changed. They do not tell you the system got better. Jodie treats LLM evaluation as a measurement problem, one psychology has worked on for over a century, and applies the psychometric idea of validity: not only "is this score reliable?" but "what evidence supports the conclusion I want to draw from it?" She starts with how benchmarks fail basic validity checks, moves to the metrics teams build for their own applications, such as "helpfulness" or "task success", and closes with four validity questions to apply to your next eval.

​What one line of Code Mode does to your agent

​Laís Carvalho, Developer Relations at Pydantic

​If you've tried writing your own agents, you've probably tried a weather agent. My Pydantic AI weather agent has two tools, get_lat_lng and get_weather, and it does fine when I ask about two locations. So I gave it a harder question: which EU capital will be warmest this weekend? That's 27 cities and 54 tool calls, with every result going back through the model. It got there, slowly. In one of my test runs it decided 8 capitals were probably enough.

​In this talk we'll add Code Mode to that agent. The model will write a short Python script that loops over all 27 cities. The script runs in Agent Sandboxes, the hosted Monty sandboxes in Pydantic Logfire, where it can only call the functions I hand it. Let's compare the runs in Logfire: tokens, cost, time, and latency using a sandbox.

​From Company Knowledge to Living Skills: GuruSup's Journey with Pydantic AI and Monty

​Fran Pastor Agustí, CTO at GuruSup

​Fran walks through an agent that investigates a customer’s order sync incident across Gmail, MongoDB and Logfire, then records it in Notion. Both agents are built with Pydantic AI and return typed Pydantic results. The talk follows how the skill evolves. It starts as a Markdown procedure that guides the tool calls one by one. Then Code Mode lets the model write Python that chains the calls, running in the Pydantic Monty sandbox. A second agent, the improver, writes one program for the whole investigation, which is saved in the skill so later runs just execute it. When it fails, the agent falls back to the Markdown procedure and still finishes the task. The improver then repairs the program from the Logfire trace, and a person approves the fix as a new revision of the skill. He closes with how GuruSup does this behind the scenes, turning company knowledge into executable skills without users having to build and refine the workflow themselves.

​Schedule

  • ​18:00 Doors open

  • ​18:30 Talks and Q&A

  • ​19:30 Food, drinks and networking

  • ​21:00 Close

​Who should come

​Python developers, ML engineers and anyone building with LLMs who wants to hear what happens after the demo. Spots are limited to 100.

Location
Please register to see the exact location of this event.
Berlin, Germany
Avatar for PyAI
Presented by
PyAI