vLLM Taipei Meet Up

​Organized by EmbeddedLLM · Co-organized by IMA (Information Management Association, Taiwan) | TAIONE OPEN SOURCE FOUNDATION

​主辦單位： EmbeddedLLM · 協辦單位： IMA 資訊經理人協會 | 財團法人開源基金會

​Event Overview 活動簡介

​An afternoon dedicated to open-source LLM inference, bringing together vLLM's core maintainers, industry practitioners, and Taiwan's AI community. Talks span core architecture, GPU optimization, real-world deployment, evaluation, and local-language model adaptation.

​Come learn, exchange ideas, and network directly with the people building the tools — and help strengthen Taiwan's open-source AI inference ecosystem.

​聚焦開源 LLM 推論的技術午後，邀集 vLLM 核心維護者、業界實務專家與台灣 AI 社群，分享核心架構、GPU 優化、實際部署、模型評測，以及在地化語言模型調適等主題。歡迎前來學習交流，直接與打造這些工具的人才互動，一同壯大台灣的開源 AI 推論生態系。

​Agenda

​14:00 – 14:10 | Registration 報到入場

​14:15 – 14:35
Introduction to vLLM: High-Performance LLM Serving & Project Update | Tun Jian, Tan 陳敦健 (TJ) | Core Contributor of vLLM, Sr AI Engineer @ Embedded LLM

​vLLM is the most popular engine for high-performance LLM serving. In this talk, we'll introduce the core architecture and features that make Large Language Model (LLM) inference and serving fast, easy, and cost-effective for everyone. Whether you are new to the engine or running production workloads, join us to explore the latest feature updates.

​vLLM 是目前最受歡迎的高效能 LLM 服務引擎。在這場分享中，我們將介紹讓大型語言模型（LLM）推論與服務變得快速、易用且具成本效益的核心架構與功能。無論您是剛接觸 vLLM，還是已在正式環境中運行相關服務，都歡迎加入，一同了解最新功能更新。

​14:35 – 14:55
vLLM x Kimi K3 & vLLM Q3 Roadmap | Tiezhen Wang | Inferact

​In this talk, we'll walk through the journey of adding vLLM support for Kimi K3 — from architecture differences and integration challenges to the optimizations needed to serve it efficiently. We'll also give an exclusive preview of the vLLM Q3 roadmap, covering what's coming next for the project and what the community can look forward to in the months ahead.

​要讓一個新的前沿模型在 vLLM 上跑起來，背後需要哪些工程？這場分享將帶我們回顧為 vLLM 加入 Kimi K3 支援的完整歷程——從架構差異、整合挑戰，到讓它高效運行所需的最佳化工作。同時，我們也將搶先公開 vLLM Q3 的產品路線圖，帶大家看看這個專案接下來的发展方向，以及社群在未來幾個月可以期待的功能與更新。

​14:55 – 15:15
From Open Weights to Local Intelligence: Practical Paths to Local LLMs | Dr. Huang, Hen-Hsen 黃瀚萱博士 | The Institute of Information Science, Academia Sinica 中央研究院資訊科學研究所

​Building a trustworthy local LLM is not simply a matter of adding local data to an open model. Local teams face limited compute, legally restricted corpora, difficult multimodal data collection, and the lack of stronger localized models for data generation and evaluation. Continual pre-training may also require costly realignment to preserve general capabilities. This talk presents practical ways to address these challenges through legal data sharing, Taiwanese Chinese evaluation, lightweight adaptation methods, and strategic use of increasingly capable open models. The goal is to show how local developers can create meaningful value without reproducing the entire frontier-model training pipeline.

打造一個值得信賴的在地化 LLM，並不僅僅是在開源模型上加入在地資料這麼簡單。在地團隊往往面臨運算資源有限、法規限制下可用語料受限、多模態資料蒐集困難，以及缺乏更強大的在地化模型可用於資料生成與評估等挑戰。持續預訓練（continual pre-training）也可能需要付出高昂的重新對齊成本，以維持模型的通用能力。這場分享將提出實務上的因應之道，包括合法資料共享、繁體中文評測、輕量化調適方法，以及策略性地運用日益強大的開源模型。目標是展示在地開發者如何在不需重現整套前沿模型訓練流程的情況下，創造出真正有意義的價值。

​15:15 - 15:25 ☕ Mid Break 中場休息

​15:25 - 15:45
Twinkle Eval × vLLM: Efficient Evaluation & High-Throughput Inference | Teds Lin 林文彬 | Twinkle AI.

​How can we efficiently compare the capabilities of different large language models while maximizing inference performance with limited hardware resources?

​This session introduces how Twinkle Eval enables a standardized and automated LLM evaluation workflow, and how it can be combined with vLLM to achieve high-throughput inference. We will cover vLLM's parallelization strategies (TP/PP/EP), batching and scheduling mechanisms, and practical techniques for optimizing deployment performance. Through real-world case studies, we'll also share best practices for benchmarking, evaluation pipeline design, and discuss the actual impact of quantization on Traditional Chinese performance, helping developers build reproducible, measurable, and maintainable LLM development and deployment workflows.

​本場分享將介紹 Twinkle Eval 如何實現標準化、自動化的 LLM 評測流程，並說明如何結合 vLLM 達成高吞吐量推論。內容涵蓋 vLLM 的平行化策略（TP/PP/EP）、批次處理與排程機制，以及部署效能優化的實務技巧。我們也將透過實際案例分享，說明基準測試（benchmarking）與評測流程設計的最佳實務，並探討量化（quantization）對繁體中文效能的實際影響，協助開發者打造可重現、可衡量、易維護的 LLM 開發與部署流程。

​15:45 – 16:05
Doing More with Less: Maximizing LLM Performance on Limited GPUs | Kuan Fu, Liu | Core Contributor of vLLM, Sr AI Engineer @ Embedded LLM

​How do you serve larger models, longer contexts, and more users when GPU memory is limited? This session explores practical techniques to push vLLM further using DFlash, KV-cache compression, KV offloading, sleep mode, prefix caching, chunked prefill, and smarter batching. We'll break down when each optimization helps, what trade-offs it introduces, and how to tune for throughput, latency, and memory efficiency. The goal: get more performance from every GPU you already have.

​當 GPU 記憶體有限時，該如何服務更大的模型、更長的上下文，以及更多使用者？本場分享將探討如何運用 DFlash、KV cache 壓縮、KV offloading、sleep mode、prefix caching、chunked prefill 以及更聰明的批次處理（batching），進一步發揮 vLLM 的效能。我們將逐一拆解每項優化技巧的適用時機、其帶來的取捨，以及如何針對吞吐量、延遲與記憶體效率進行調校。目標只有一個：從您現有的每一張 GPU 上榨出更多效能。

​16:05 - 16:25
Breaking the Memory Wall: Extending vLLM's KV Cache with WEKA's Augmented Memory Grid | 吳岱侑 Ray Wu | Sr Technical Consultant 亞太區資深技術顧問 @ WEKA

​KV cache eviction is a silent killer for long-context vLLM workloads — forcing redundant prefill and tanking latency. We'll show how WEKA's Augmented Memory Grid turns NVMe into a microsecond-latency KV cache extension via an open-source GDS integration for LMCache, with benchmarks showing up to 20× faster Time to First Token at 128K context.

​在長上下文的 vLLM 工作負載中，KV cache 被迫淘汰（eviction）往往是效能的隱形殺手——導致重複的 prefill 運算，並嚴重拖累延遲表現。這場分享將展示 WEKA 的 Augmented Memory Grid 如何透過開源的 GDS 整合方案，將 NVMe 轉化為微秒級延遲的 KV cache 延伸儲存層（結合 LMCache），並展示在 128K 上下文長度下，首字回應時間（Time to First Token）最高可提升達 20 倍的實測數據。

​16:25 - 16:55 Closing & Group Photo & Networking 閉幕 & 合影 & 自由交流

​

​Please note: The agenda, topics, and speakers are subject to change without prior notice due to unforeseen circumstances. 本活動議程與主講嘉賓可能因突發狀況有所調整，主辦單位保留最終修改與變更之權利。