LLM-D 101: Distributed AI Inference on Kubernetes
As large language models become part of modern applications, running them efficiently at scale introduces new challenges for cloud-native infrastructure.
In this introductory session, we'll explore LLM-D, an open-source CNCF Sandbox project designed to support distributed LLM inference on Kubernetes.
We'll start with the basics of LLM inference, understand the challenges of traditional model serving, and introduce how LLM-D uses intelligent request routing, KV caching, and distributed serving to improve resource utilization and performance.
The session will also explore how familiar cloud-native technologies such as Kubernetes, Gateway API, and vLLM fit into the LLM-D architecture.
Who should attend: DevOps engineers, platform engineers, Kubernetes enthusiasts, and anyone interested in AI infrastructure.
Prerequisites: Basic Kubernetes knowledge is helpful. No prior AI or machine learning experience is required.