Engineering GPU Infrastructure for AI Workloads
Most Platform and DevOps engineers know how to deploy an application. Far fewer know what changes when that application needs a GPU.
This hands-on workshop takes platform and DevOps engineers from Kubernetes basics to working with GPU infrastructure for AI inference on Amazon EKS. You will see why GPUs behave differently from ordinary CPU and memory resources, how Kubernetes discovers and schedules them, and how the NVIDIA stack fits together: drivers, container runtime, device plugin, GPU Operator, and GPU-aware node provisioning.
The session is deliberately focused on inference, not model training. You will serve a real LLM using vLLM on GPU nodes, expose it through a service, test the endpoint, load it, autoscale it, and monitor what is happening underneath. Along the way, you will work with the operational pieces that matter in real teams: Karpenter NodePools, EC2NodeClass, GPU-optimized AMIs, taints and tolerations, NVIDIA GPU Operator, time-slicing, HPA, DCGM metrics, Prometheus, Grafana, Spot capacity, and teardown discipline.
This is not a theoretical “AI infrastructure” overview. You will see where GPU workloads fail in production: quota issues, driver mismatches, idle GPU burn, slow image pulls, cold starts, model load times, poor bin packing, and scheduling patterns that make sense for CPUs but break down with expensive accelerators.
By the end, you will have taken a cluster from zero GPUs to serving a live LLM on GPU-backed Kubernetes infrastructure, with autoscaling, sharing, monitoring, and cost controls in place. You will also leave with a companion repo so you can rerun the workshop, inspect every manifest, and adapt the pattern for your own AI platform work.
The live build
You will build a GPU-ready inference platform on Amazon EKS. The build starts by adding GPU capacity to an EKS cluster using Terraform, Karpenter, a GPU-aware NodePool, and an EC2NodeClass for G-family instances such as g5/g6. You will install the NVIDIA GPU Operator with the right EKS configuration, validate the setup with nvidia-smi, and confirm that Kubernetes can see nvidia.com/gpu as an allocatable resource.
Then you will move into GPU scheduling and sharing. You will apply a time-slicing configuration, run multiple inference pods on one GPU, and watch Karpenter provision another GPU node when demand exceeds available capacity.
The main build is a live LLM inference service using vLLM. You will deploy the model, expose it through a Kubernetes Service, test the OpenAI-compatible endpoint with curl, load test it with hey or k6, and observe autoscaling as new GPU capacity joins the cluster.
Finally, you will inspect GPU utilization, memory, temperature, throttling, and per-pod attribution using DCGM, Prometheus, and Grafana.
Why now
AI applications are moving from demos into production systems, and that changes what platform teams are expected to support. It is no longer enough to know Kubernetes for stateless web apps, CI jobs, and microservices. The next wave of platform work involves LLM inference, GPU nodes, accelerator scheduling, model-serving latency, GPU observability, and cost-aware autoscaling.
GPU infrastructure is also unforgiving. A CPU node sitting idle is wasteful; a GPU node sitting idle is expensive. A bad AMI choice can cause driver mismatches. A narrow instance-type selection can leave workloads pending when Spot capacity disappears. A model may take minutes to become useful because the node has to boot, the image has to pull, and the model has to load.
That is why GPU infrastructure is becoming a practical career skill for platform engineers, DevOps engineers, SREs, and Kubernetes operators. The teams that can serve models reliably, scale them intelligently, and control GPU spend will be the teams that make AI usable in production.
You will walk away with
A working mental model of the GPU stack on Kubernetes: NVIDIA driver, container toolkit, device plugin, GPU Operator, and EKS GPU-optimized AMIs
Hands-on experience building a GPU-ready Amazon EKS cluster with Terraform and Karpenter
A clear understanding of GPU scheduling, including nvidia.com/gpu, integer GPU requests, taints, tolerations, node selectors, and affinity
Practical guidance on GPU sharing patterns: exclusive GPUs, time-slicing, MIG, and MPS, with trade-offs for cost, isolation, and latency
A live vLLM inference deployment serving a real open model on GPU nodes
Experience testing an OpenAI-compatible inference endpoint using curl and load-testing tools
A working view of autoscaling GPU workloads with HPA and Karpenter
A practical GPU observability checklist using DCGM exporter, Prometheus, and Grafana
Cost-control tactics for GPU fleets: Spot diversification, AMI pinning, autoscale-to-zero, right-sizing, bin packing, and teardown discipline
A companion repo with Terraform, Helm values, manifests, dashboards, validation scripts, and make workshop-up / make workshop-down
Why this session is different
Most AI infrastructure sessions either stay too high-level or assume you already know the GPU stack. This workshop sits in the gap that platform engineers actually face: you know Kubernetes, but GPUs still feel like a separate world.
Here, the GPU is not an abstract accelerator on a slide. You will see how it appears inside Kubernetes, why nvidia.com/gpu: 1 behaves differently from CPU and memory requests, why fractional GPU allocation is not native, and how time-slicing, MIG, and MPS fit into real scheduling decisions.
The session also treats cost and operations as first-class engineering concerns. You will not just deploy a model and stop there. You will watch what happens when pods are pending, when Karpenter has to provision a GPU node, when cold starts slow the system down, when utilization is poor, and when sharing can improve the economics.
It is also honest about scope. This is an inference-serving workshop, not a distributed training workshop. Training topics such as EFA, NCCL, DRA, and multi-cluster GPU fleets are called out in the wrap-up, but the hands-on work stays focused on the production pattern most platform teams need first: serving AI workloads reliably on Kubernetes.
Who should attend
This workshop is for platform engineers, DevOps engineers, SREs, cloud engineers, Kubernetes operators, and infrastructure engineers who are comfortable with Kubernetes basics and want to move into AI infrastructure.
You are a strong fit if you can work with pods, deployments, services, kubectl, and basic Kubernetes troubleshooting, but are new to GPU nodes, NVIDIA tooling, model-serving infrastructure, and GPU cost management.
It is especially useful for engineers who are being asked to support LLM inference, internal AI platforms, model-serving environments, GPU-backed Kubernetes clusters, or production AI workloads on AWS.
This is not designed for data scientists looking for model training theory. It is for infrastructure engineers who need to make AI workloads run reliably, securely, observably, and cost-effectively.
Prerequisites
Comfort with Kubernetes basics: pods, deployments, services, and kubectl
A laptop with kubectl, Helm, AWS CLI, and Terraform installed
Setup guide completed before the workshop
Access to an AWS account with EKS permissions, or credentials to the shared workshop account if provided
GPU quota for G-family instances such as g5/g6 in the workshop region, approval can take 24 - 48 hours, so request it at least a week before the workshop.
Basic command-line familiarity
No prior GPU infrastructure experience required
No model training experience required