Real-Time AI: Streams, Agents & SQL Indexing with Elastic + Kafka
Join the Elastic Chicago User Group and Confluent on Wednesday, May 13th for a meetup. We'll have presentations from followed by food, refreshments, and networking.
Thank you to Improving for hosting our meetup. Improving is a modern digital services consulting firm focused on solving business problems with innovative technical solutions.
RSVP Instructions:
Please provide your email when you register for this event. Please bring an ID to the event for check-in.
📅 Date and Time:
Wednesday, May 13th from 5:30-8:00 pm
📍Location:
Improving - 222 S Riverside Plaza 15th Floor, Chicago, IL 60606
🔑 Arrival Instructions:
Please bring an ID for check-in
Upon arrival at 222 Riverside, check in at the Improving table
Take the elevator to the 15th floor - the meetup will be in the common area through the doors.
🚗 Parking:
SpotHero is a great option to resvere parking in advance
Improving recommends the following parking garages: 229 S. Desplaines St (which is also the 625 W. Adams building), 500 West Monroe, and the outdoor lot at 718 W Monroe.
📝Agenda:
5:30 PM – Doors open, grab a seat, and enjoy some pizza!
6:00 PM – Welcome & Introduction by Improving
6:05 pm – One Does Not Simply Query a Stream - Viktor Gamov, Principal Developer Advocate, Confluent
6:30 – Building Autonomous AI Agents with Elasticsearch & LLMs (Lightning talk) - Deepesh Kumar (MS in AI @Illinois Tech)
6:45 pm – A Practical Approach to Real‑Time SQL Data Indexing in Elasticsearch - Mangesh Walimbe (Application Architect)
7:00 pm: Networking and refreshments
8:00 pm: Event ends
💭 Talk Abstracts:
One Does Not Simply Query a Stream - Viktor Gamov, Principal Developer Advocate, Confluent
Streaming data with Apache Kafka® has become the backbone of modern applications. While streams are ideal for continuous data flow, they lack built-in querying capabilities. Unlike databases with indexed lookups, Kafka’s append-only logs are designed for high-throughput processing—not for on-demand queries. This necessitates additional infrastructure to query streaming data effectively.Traditional approaches replicate stream data into external stores: relational databases like PostgreSQL for operational queries, object storage like S3 accessed via Flink, Spark, or Trino for analytics, and Elasticsearch for full-text search and log analytics. Each serves a purpose—but they also introduce silos, schema mismatches, freshness issues, and complex ETL pipelines that increase system fragility.In this session, we’ll explore solutions that aim to unify operational, analytical, and search workloads across real-time data.
We'll demonstrate stream processing with Kafka Streams, Apache Flink®, and SQL engines; real-time analytics with Apache Pinot® ; search capabilities with Elasticsearch; and modern lakehouse approaches using Apache Iceberg® with Tableflow to represent Kafka topics as queryable tables. While there's no one-size-fits-all solution, understanding the tools and trade-offs will help you design more robust and flexible architectures.
A Practical Approach to Real‑Time SQL Data Indexing in Elasticsearch - Mangesh Walimbe (Application Architect)
SQL databases are designed to store and manage large volumes of structured data, but they are not optimized for high‑speed search and analytics. Elasticsearch, by contrast, provides powerful indexing and near real‑time search capabilities, making it a strong complement to traditional relational systems. Ensuring that SQL data is continuously available in Elasticsearch, however, can be a complex and time‑consuming process. This presentation outlines a streamlined approach for processing, filtering, and ingesting SQL database records into Elasticsearch using Logstash pipelines, based on practical implementation experience.