(Apologies for Mistake) Real Practical RAG, from Demo to Production at the Scale of Thousands
Apologies to everyone for this mistake.
This was originally a private event draft that was being edited by the Speaker.
The actual event page is over here: https://luma.com/sosi6fei
Sincere apologies that an invite was accidentally sent out on this event instead of the actual event page
Please register over at the actual event page https://luma.com/sosi6fei instead of here, thank you.
The talk that was being drafted:
Real Practical RAG, from Demo to Production - in a 10,000-person organization and an intelligence org
Moving from controlled demos to real-world deployment exposed several challenges that are easy to underestimate. In clean demos, our prompts/queries are often well-formed and datasets are neatly curated, and we test the happy paths.
But real users behave very differently - there are gaps between how users express intent and how that intent is represented in the embedding space. This made retrieval design and chunking strategy critical: how documents are split, indexed, and ranked directly impacted whether the system could “understand” a query.
Additionally, as the dataset scaled with many similar documents, retrieval became noisier - every vector is similar to a lot other vector - making it harder to consistently surface the right information.
These challenges highlight that building a production-ready RAG system is not just about ensuring our datasets and RAG method works, but also user's intent and behavior.
Key takeaways from the talk:
1) RAG systems: user behavior and prompt clarity matter as much as model design
2) Hybrid retrieval (vector + keyword) is essential for balancing semantic understanding and precision
3) Scaling data introduces noise, requiring better chunking and retrieval strategies
4) Production success comes from continuous iteration based on real user behaviors, not just a good demo
---
TBA: We will have a 2nd speaker talking about AI in real use cases in intelligence orgs either in this meetup or the next meetup. More details for the 2nd speaker are still being prepared.
The 2nd speaker will customize the talk to be relevant for both non-technical people (from a management decision POV) and technical people (from an implementation POV).
---
Special Thanks:
- To SQ Collective for the venue
- This is the SQ Collective calendar that you can subscribe to: https://luma.com/Ai-labs
- This is the Technology calendar that you can subscribe to: https://luma.com/calendar/cal-ZrFfXqC7PgzbBaQ - It will include tech events from tech unicorn startups such as Amplitude, Databricks, Elastic, MongoDB, and more