Cover Image for Evaluating and Improving Agent Search Loops

Presented by

The MLOps Community fills the swiftly growing need to share real-world Machine Learning Operations best practices from engineers in the field. While MLOps shares a lot of ground with DevOps, the differences are as big as the similarities. We needed a community laser-focused on solving the unique challenges we deal with every day building production AI/ML pipelines.

We’re in this together. Come learn with us in a community open to everyone. Share knowledge. Ask questions. Get answers.

Join us every week for a live virtual meetup, Wednesdays at 5 PM UK, 9 AM PT, where top experts share what they’ve learned running pipelines for organizations big and small. We’re also firing up “coffee sessions” where we dive deep into the emerging stack of ML tools and platforms.

You can also check out our Slack and podcast that’s filled with tips and tricks to overcoming the common obstacles we’ve all hit in the real world. Find the solutions you need. Share, learn and grow with us, as we work to bring standardization to the chaotic world of MLOps.

Hosted By

1 Going

AI

Evaluating and Improving Agent Search Loops

Name: Evaluating and Improving Agent Search Loops
Start: 2026-06-10T17:00:00.000-07:00
End: 2026-06-10T21:00:00.000-07:00
Location: Plug and Play Tech Center

San Francisco MLOps Community

Plug and Play Tech Center

Sunnyvale, CA

Approval Required

Your registration is subject to host approval.

Welcome! To join the event, please register below.

You will be asked to verify token ownership with your wallet.

About Event

Most retrieval agents fail quietly. The agent pulls something plausible, generates an answer, and only later do you realize the context was incomplete or wrong.

In this hands-on workshop, you'll build and evaluate an agent that recognizes when its retrieval step isn't good enough. Starting from a prebuilt Qdrant-powered agent, you'll run evals, inspect traces, identify failure modes, and add in-loop checks that help the agent decide whether to rewrite the query, search again, or stop.

We'll cover practical retrieval-quality signals: low top-result confidence, tightly clustered scores, weak rank gaps, and disagreement between keyword and vector retrieval. Then we'll use those signals to improve the loop and measure whether the changes actually improve answer quality, retrieval quality, cost, and time-to-success.

No pre-work required. Every participant gets a ready-to-use VM with the environment, data, agent code, Qdrant setup, traces, and evaluation harness included.

AGENDA

5:00 – 5:30 PM | Arrival, Registration & Networking
Check-in, refreshments, and access to the demo environment

5:30 – 6:00 PM | Introduction, Opening

6:00 – 6:20 PM |

6:20 – 7:20 PM | Talk: TBC

7:20 – 7:50 PM | Talk: TBC

7:50 – 8:20 PM | Talk: TBC

8:20 – 8:50 PM | Wrap-Up & Networking
Key takeaways, Q&A, and networking

What You’ll Learn

How to evaluate retrieval agents beyond final answer correctness
How to inspect traces and identify retrieval-driven failures
How agents can use live retrieval signals to decide when to retry
How to detect weak retrieval using score confidence, variance, rank gaps, and retriever disagreement
How query rewriting can improve or harm retrieval
How to measure whether step N+1 actually beats step N
How to balance answer quality, latency, token cost, and tool-call cost
How Qdrant serves as the retrieval layer for adaptive agent systems

WHO SHOULD ATTEND

AI engineers and agent developers who want to move beyond "retrieve and generate" and learn how to evaluate and improve agent behavior step by step. Best fit for people who have built basic agent workflows and want stronger tools for debugging retrieval failures, query rewriting, and stopping decisions.

Seats are limited

Location