Monitoring LLM Applications: Traces, Feedback, and Production Quality
This is the 5th workshop in our series to update the LLM Zoomcamp content.
This workshop updates Module 5: Monitoring.
In this hands-on session, Alexey Grigorev will show how to monitor an LLM and RAG application after it is deployed.
You’ll learn how to instrument an LLM pipeline, collect traces and metrics, store chat history, track user feedback, and evaluate answer quality in a running system.
What you’ll learn:
Why monitoring matters for LLM and RAG applications
What to monitor in a deployed LLM system
How to collect traces from an LLM pipeline
How to instrument LLM calls and retrieval steps
How to store chat history and user interactions
How to collect user feedback on generated answers
How to track answer quality over time
How to detect issues in retrieval and generation
How to use dashboards to inspect system behavior
How to run evaluations on production traces
How monitoring connects offline evaluation with real-world usage
By the end, you’ll understand how to monitor a deployed RAG application and inspect what happens after users start interacting with it.
Like the other workshops, this will be a live demo with practical tips and time for Q&A.
All events in these series:
Vector Databases: Embeddings, Semantic Search, and Hybrid Retrieval
RAG and Agents Evaluation: Measuring Retrieval and LLM Answer Quality
Monitoring LLM Applications: Traces, Feedback, and Production Quality
Thinking about Joining LLM Zoomcamp?
This workshop covers the updated content for Module 5 of the LLM Zoomcamp, our free course on building practical LLM applications with RAG, vector search, evaluation, monitoring, and AI agents.
You start with a simple RAG pipeline, then improve it with better retrieval, semantic search, function calling, evaluation, monitoring, and production practices.
The course covers the full lifecycle of an LLM application: from the first working prototype to evaluation, monitoring, and a complete final project.
The new cohort of LLM Zoomcamp starts on June 8, 2026. You can join it by registering here.
About the Speaker
Alexey Grigorev is the Founder of DataTalks.Club and creator of the Zoomcamp series.
Alexey is a software and ML engineer with over 10 years in engineering and 6+ years in machine learning. He has deployed large-scale ML systems at companies like OLX Group and Simplaex, authored several technical books, including Machine Learning Bookcamp, and is a Kaggle Master with a 1st place finish in the NIPS’17 Criteo Challenge.
DataTalks.Club is the place to talk about data. Join our Slack community!