Near Real-Time Retrieval-Augmented Generation (RAG)

​Near Real-Time Retrieval-Augmented Generation (RAG) Pipelines at Scale in Java

RAG is the pattern used to improve LLM accuracy and minimize AI hallucinations. But most implementations are a patchwork of batch embedding jobs, external vector databases, message brokers, miscellaneous services, and brittle glue code. The result is predictable: higher latency, tougher and more costly operations.
In this talk, you’ll see a streaming RAG architecture built natively in Java: continuous ingestion and transformation with distributed DAG pipelines, horizontally scalable embedding inference, and in-memory distributed vector collections for millisecond semantic search. We’ll show how data can be vectorized as it arrives from CDC, events, REST sources, and documents, how partition-aware processing cuts network overhead, and how co-locating compute with vector storage enables fast retrieval with filtering and enrichment in a single runtime. The goal is a unified, stateful platform that reduces architectural sprawl while improving latency and resilience.

​Key Discussion Points:

• ​Why most enterprise RAG systems are not truly real-time

• ​Trade-offs between batch, micro-batch, and streaming AI architectures

• ​How streaming pipelines enable continuous ingestion and embedding

• ​Architectural patterns for distributed ML inferencing at scale

• ​How in-memory vector collections support resilient low-latency semantic search
  
  Speaker - Scott McMahon

​Scott McMahon is the Senior Director of Solution Architecture at Hazelcast®, with over 25 years of experience in software development and enterprise consulting. Before joining Hazelcast, he built big-data analytics platforms and business-process management systems for many of the world's leading corporations. He currently lives in Portland, Oregon, and when not working on computer systems, he enjoys getting outdoors and having fun with his family.