PROBE Evaluation Design Jam
The Gap Between Capability and Verified Safety is Widening
Agentic AI systems are shipping faster than we can evaluate them. Meanwhile, regulators across the EU, US, Singapore, and China are demanding documented adversarial testing. Most teams still don't have the tooling or methodology to meet that bar.
PROBE is a hands-on evaluation design jam where builders and researchers who care about AI safety + security come together to share and co-design evaluation methods. PROBE is designed as a discussion x working session for sharing benchmark datasets, stress-test harnesses, and open-source tooling that the community can reference as they evaluate agents.
The goal is to connect with other builders and researchers interested in AI safety and assurance - and create a working group for AI assurance.
Curated by AI Safety Node/Ecomonitor (www.aisafetynode.com) and hosted with Ojin(https://ojin.ai/).
**** Program and Working Group ****
Talks: Human in the Loop x AI Evaluation, Benchmark Exploring Edge Cases in AI Alignment
Humans in the Loop x AI Evaluation
Trevor Lohrbeer is a co-founder of AI Safety Berlin, a coworking & community hub, for those working to reduce societal risks from AI in Berlin, and an independent AI safety researcher focused on keeping humans in the loop during recursive self-improvement. Prior to going independent, he spent 16 months working with Redwood Research developing AI control evaluations and doing vulnerability research on Claude Code auto mode, and spent 25 years founding and running Internet and software startups. Learn more about his AI safety work here.
A Benchmark exploring Edge Cases in AI Alignment
Florian Dietz is an AI alignment researcher and consultant. He will share his work on creating benchmark of scenarios that are Out-Of-Distribution (OOD) for current AI systems
Florians' Linkedin: floriandietz
Elective Design Sprint Sessions
Safeguard Stress Testing Design
Share and work on evaluation design that probe guardrails, test refusal mechanisms and results (if past evaluations are available)
Evaluation Tools + Infra
Discuss evaluation tools and infrastructure. Share and work on reusable harnesses, metrics pipelines, and reporting tools that make adversarial testing repeatable, not one-off.
Schedule
18-19:00: Kick off and guest speaker on evaluation method
Speakers from AI Safety Hub in Berlin and independent research firms to discuss evaluation methods and red teaming x agentic harnesses.
19:00 - 20:00: Evaluation design: Eval interactive judging game or mixer
20:00-20:30: Share eval design work through standup and researcher exchange
This event is open to all. Safety researchers building adversarial benchmarks and agent builders/ML engineers who are exploring eval tooling will find the evening especially relevant.
******* About Ojin*******
Ojin AixHaus is the new AI Playground in Berlin. A collaborative community space where scientists, machine learning engineers, founders, creators, and AI enthusiasts come together to build the future of AI.
Ojin AixHaus is free to join, open for startups in residency applications and are currently curating its 2026 events calendar. If you have a burning idea for an event, meetup or hackathon - bring your fire, they bring the space!
Find out more here:
Ojin AixHaus Community
Follow Ojin AixHaus on socials:
Instagram: ojin.ai
Linkedin: https://www.linkedin.com/showcase/ojin-aixhaus/?viewAsMember=true
X: Ojin_AI
Ecomonitor/AI Safety Node:
AI Safety Node, an initiative by Ecomonitor, bring researchers and product developers together for AI safety/security development and community initiatives. Ecomonitor is a research studio focused on complex risks. Read more about our work here:
LinkedIn:
ecomonitor
AI Safety Node Initiative:
www.aisafetynode.com