Evals #4, Miton AI Times edition
Three sold-out editions later, evals.cz is back for round four, this time together with the Miton AI Times community.
The premise stays. Model benchmarks are everywhere, but what you care about is different: does it work for you, on your data, for your users? evals.cz is the Prague meetup for people who build AI products and need to measure them.
Evelina Gabasova (AISLE) opens with the keynote. Antonín Hoskovec (Miton, anex.sh) and Michal Spiegel (Filevine) follow.
What to expect
3 short talks
practical experience from production
no sales pitches
talks in English
Schedule:
17:00–17:15 Doors open & registration
17:15–17:20 Intro
17:20–17:40 Evelina Gabasova: The Weird and Wonderful World of Cybersecurity Evals
17:40–17:45 Q&A
17:45–18:05 Antonín Hoskovec: Evaluating Open Models on Your Task and in Your Language
18:05–18:10 Q&A
18:10–18:20 Break
18:20–18:40 Michal Spiegel: Generating Synthetic Evaluation Data for Search Agents
18:40–18:45 Q&A
18:45–19:00 Quiz
19:00–21:00 Networking
Evelina Gabasova: The Weird and Wonderful World of Cybersecurity Evals
AI cybersecurity evals have suddenly become a mainstream concern, and the OpenAI/Hugging Face incident shows benchmarks are now adversarial environments. In this talk, we take a practical view of evaluating models for vulnerability discovery and remediation. Using examples from AISLE’s work, we’ll discuss the problems with public benchmarks, challenges in building our own, and failure modes when the eval itself is the attack surface.
Antonín Hoskovec: Evaluating Open Models on Your Task and in Your Language
Every new open-weight release raises the same questions: is it better on my task, is it better in my language, and could a smaller model do the job? I'll show how I answer them, using Czech as the example. For instance I use calibrated LLM judges on task-specific data to find the smallest model that is still sufficient, and I evaluate models on BenCzechMark and a Czech localisation of τ²-bench for agentic tool use. The whole pipeline can run in GitHub Actions, so a model swap, prompt change, or inference-stack update is tested like any other code change and regressions are caught before deployment.
Michal Spiegel: Generating Synthetic Evaluation Data for Search Agents
Evaluating search agents is hard, manual annotation is expensive and often infeasible. Synthetic generation scales, but it is easy to produce questions that no real user would ask. This talk describes how we generate synthetic evaluation data for search agents that answer questions over a large set of case files. Humans stay in the loop throughout: they steer the generation and review the results manually. The talk covers how we generate deep research QA data, as well as how we measure and control their quality and diversity.
Who it's for
ML/AI engineers, backend developers integrating LLMs, data scientists, product managers, AI researchers, and curious beginners.
This time Miton hosts us at their Truhlárna space, and we're running the event jointly with Miton AI Times, so you'll meet people from both communities.