delta-explain: Making Delta Lake File Pruning Visible, and Testable in CI
For scan-heavy lakehouse workloads, one of the biggest performance and cost levers is decided before any engine runs: whether a file can be eliminated before it is ever opened. That decision lives in the table's physical layout and metadata, and it is normally invisible. A query can look perfectly reasonable while the layout has already made it expensive. Every file you don't skip is bytes you scan and money you pay.
This session introduces delta-explain, a small open-source CLI that makes Delta Lake file pruning visible, measurable, and assertable. Built on delta-kernel-rs, it doesn't reimplement pruning: the kernel computes the surviving file set correctly, and delta-explain attributes the result to its two distinct mechanisms (partition pruning and data skipping) and explains it per file, all from transaction-log metadata, without executing the query in any engine.
We'll run it live on the same logical query over two physical layouts, watch the pruning collapse, then turn the same analysis into a CI contract: a threshold and an exit code that fails a pipeline before a regression reaches production and the bill.
Speakers
Christian Del Monte is a Senior Software Architect at adesso SE. He builds and works on data-lakehouse systems and is the author of the open-source delta-explain.
Robert Pack began his career within a multinational chemical company, where he designed and led transformative cloud and data initiatives. As Chief Technology Lead, he was responsible for global data and AI platform engineering.
Robert’s open-source journey began with contributions to Delta Lake and Delta-RS and later expanded into the broader Rust and Apache Arrow data ecosystems. Today, as part of Databricks, he serves as a Developer Advocate dedicated to fostering a modern, high-performance data ecosystem for all. Through his work on projects such as Delta Kernel, Delta Lake, Unity Catalog, and Apache Iceberg, Robert remains committed to advancing the Open Lakehouse vision and facilitating the next generation of data infrastructure.