DataFusion Community Showcase Vol.2

​We're excited to present the second DataFusion Community Showcase!

​This virtual session will highlight two projects building on DataFusion:

​Jordan Epstein, DataFusion Comet Iceberg Writes

​"Accelerating Apache Iceberg Writes and Table Maintainence with Native Code"

​Abstract: Iceberg writes and table compactions are a compute heavy workload that often goes overlooked. This talk explains the benefit of using DataFusion Comet to get the procedural logic for table compactions that lives in spark plus the performance benefit of rust and datafusion!

​Phillip Leblanc, Spice.ai

​"Building an HA, Distributed Query Engine on Apache DataFusion Ballista"

​Abstract: Apache DataFusion gives you a fast single-node engine, but some analytical and agentic workloads need to run across a cluster, survive node failures, and scale to petabytes. At Spice we built our distributed query layer on Apache DataFusion Ballista to accomplish this.

​This talk covers what it took to make Ballista highly available: stateless, object-store-native schedulers with no single point of failure, lease-based coordination and automatic failover, mTLS across all inter-node communication, and dynamic cluster sizing as executors join and leave.

​We will dig into the planner, including partition-aware routing and NDV-aware table statistics that keep large semi-joins from running out of memory, and the split between synchronous streaming and asynchronous, object-store-materialized execution. We will also show how distribution composes with local acceleration on the Vortex columnar format.