Lessons from Open Sourcing Dicer: Databricks' Auto-Sharder

​About Event

Databricks recently open-sourced Dicer, the world’s first open-source auto-sharder—a capability that, until now, has largely been confined to a small number of highly sophisticated internal systems. Auto-sharding has long enabled these systems to safely run stateful services with major performance advantages, but the operational complexity kept it out of reach for most teams.

​The common advice has been to avoid stateful designs altogether and stick with stateless services that round-trip to databases or remote caches on every request. While simpler to operate, this approach comes with higher latency and cost due to network and serialization overheads. Dicer takes a different path: it dynamically assigns ranges of keys to stateful servers, allowing hot data to live in local memory while continuously adapting to changes in load, health, and membership.

​What makes Dicer especially interesting is that it shields applications from the hardest parts of sharding. It is load-aware, minimizes unnecessary reassignment, detects hotspots, and can isolate or replicate hot keys when needed—going well beyond static approaches like consistent hashing. Inside Databricks, this model has led to dramatic reductions in database load and tail latency.

​We’re excited to host this session to learn how Dicer works, why these design choices matter, and what this means for teams that want to build fast, resilient, stateful systems without taking on unsustainable operational complexity.

​About Speakers

​Colin Meek helps build core infrastructure at Databricks supporting stateful applications. He is hopeful that platforms for stateful applications will evolve to the point that all applications can be trivially scalable, low latency, highly available, and cheap.

​Phil Bogle is an accomplished technology leader with extensive experience in software engineering and management. Bogle has held significant roles including Director of Engineering at Kalshi, Senior Manager and Senior Staff Software Engineer at Databricks, and a pivotal position as Staff Software Engineer at Google for over a decade. Bogle holds a Master’s degree from the Massachusetts Institute of Technology and a Bachelor’s degree from Dartmouth College.

​Directions to get upstairs to the meeting room

​To reach SURF Incubator on the 7th floor, locate the "21/4" elevator bank in the main lobby. Simply press the button for floor 7.

​Thank you Databricks for sponsoring the event