Exploring Delta's Rust Kernel: A behind the scenes exploration of what secretly powers delta-rs

​The Delta Lake ecosystem has evolved significantly since 2021, becoming more complex while facing fragmentation challenges across different engine implementations. The Delta Rust kernel represents a fundamental shift in how Delta Lake features are developed and deployed, centralizing core functionality to ensure consistent performance and protocol interpretation across all engines.

​Join R. Tyler Croy, Robert Pack, and Scott Haines in this session that explores the latest developments in the delta-rs kernel, including highly requested community features like column mapping and deletion vectors. We'll examine how the kernel addresses past architectural mistakes while providing "uniformly faster" performance gains that immediately benefit all engines utilizing it.

​While most engineers will never interact with the kernel directly (as it's designed for engine creators and not the typical end user) its impact is transformative. The kernel ensures that your Delta tables are created, updated, and managed consistently, regardless of which engine you're using, fundamentally changing how the Delta Lake ecosystem operates behind the scenes.

​Speakers

​R. Tyler Croy, Rustacean, Delta Maintainer & Buoyant Data

​Robert Pack, Staff Developer Advocate, Databricks

​Scott Haines, Developer Relations Engineer, Buf