GenAI Cracow #26 - Observability
Agenda
Opening Ceremony
Langfuse as a Management Hub for AI by Mateusz
Evals, llm-as-a-judge and building predictable systems by Konrad
Q&A
Quiz
Networking
Abstract
Langfuse as a Management Hub for AI
LLM-based applications often fail in ways that are hard to understand i.e., the model responds differently than expected, hallucinates, or produces inconsistent outputs, and it’s not clear why. Basic observability is easy to add, but making it actually useful for debugging takes more work. This talk shows how Langfuse was used beyond the default setup - how traces were structured to make debugging possible, what to log, and how to investigate real cases where the system behaved unexpectedly. We will also cover practical challenges around prompt management, including versioning, testing, and handling prompts in production (e.g. caching to avoid unnecessary latency).
Evals, llm-as-a-judge and building predictable systems
LLM-based systems fail in specific, repeatable ways starting with wrong output format, ignored rules, missing required fields. In this talk we will cover how to build evals (in Langfuse) to catch these: deterministic checks for structure and format, and LLM-as-a-judge for the cases rules can't cover. We'll go through what each eval type looks like in practice and how to read results alongside traces when something breaks. Live demo included
Speakers
Mateusz Wróblewski
Solution Architect @ Software Mind, supporting pre-sales, system architecture, and product development. Co-founder at Thyn IT, responsible for technology and architecture. Builds SaaS products, focusing on matching technology to the actual problem. Works on integrating LLM-based systems into both products and development workflows, treating AI as part of how software is built, not an add-on.
LinkedIn
Konrad Bujak
AI Engineer & Venture Builder
LinkedIn
Partners
Software Mind
Software Mind is a global technology partner with operations throughout Europe and the Americas. Driven by tech and empowered by people, we provide companies with engineers and consultants who manage software development life cycles and guide digital acceleration strategies. For over 25 years we’ve been enriching organizations with expertise that boosts scalability, drives dynamic growth and delivers customized business and technical support. A culture that embraces openness, craves more and acts with respect ensures smooth integration, meaningful cooperation and measurable value for scale-ups, unicorns and enterprise-level companies around the world.
CEE AI Hub