Apache Iceberg Virtual Meetup: GSoC Projects and Commutative Compaction

​Join us for the kickoff of the virtual Apache Iceberg meetup series:

​

​Following the Read Path: From Spark Readers to Object Storage

​w/ Varun Lakhyani

​My GSoC project began with a straightforward goal: improve Apache Iceberg's Spark reader performance through parallelism. But tracing the read path across Spark, Iceberg, Parquet, and object storage revealed the original optimization was only part of the story. Understanding those interactions and validating with benchmarks led me to a different optimization first. This session explores that investigation, the design decisions and benchmarking methodology behind it, and the performance improvements that followed, along with lessons on open source collaboration.

​

​Commutative Compaction

​w/ Chris Douglas

​Table formats optimize data layouts in immutable stores by committing compactions. Despite making no logical changes, layout changes conflict with concurrent transactions, forcing costly restarts of both compactions and user transactions. We introduce compaction maps, a compact data structure encoding remapping functions to repair layout conflicts and restore commutativity in data lake settings. We implement compaction maps in Apache Iceberg and demonstrate that repairing even large conflicts through large compactions takes less than a second, including I/O costs in three major US clouds.