DataFusion Community Showcase Vol.2

​We're excited to present the second DataFusion Community Showcase!

​This virtual session will highlight two projects building on DataFusion:

​Jordan Epstein, DataFusion Comet Iceberg Writes

​"Accelerating Apache Iceberg Writes and Table Maintenance with Native Code"

​Abstract: Iceberg writes and table compactions are a compute heavy workload that often goes overlooked. This talk explains the benefit of using DataFusion Comet to get the procedural logic for table compactions that lives in spark plus the performance benefit of rust and datafusion!

​Phillip LeBlanc, Spice AI

​"Building an HA, Distributed Query Engine on Apache DataFusion Ballista"

​Abstract: Apache DataFusion provides a fast single-node query engine, but larger analytical and agentic workloads need execution to reliably scale beyond one process. At Spice, we built our distributed query layer on Apache DataFusion Ballista.

​This talk covers why we selected Ballista and the extensions required to integrate it into a production query runtime: object-store-based cluster state, mTLS-secured communication, executor discovery and capacity management, remote catalog and UDF synchronization, configurable shuffle storage and encoding, and the reliability problems exposed at production scale.