Engineering Dynamic Lineage: Column-level lineage using Open Lineage, Spark, and Delta Lake
Traditional static lineage tools fail to track real-time data flow in complex enterprise environments. This session explores dynamic, deterministic lineage as telemetry—treating query execution as parseable events that reveal current data state. We'll examine the Open Lineage schema as an open-source alternative to proprietary vendor solutions, demonstrating how to stitch together data flows across thousands of jobs. Through live demos and architectural discussion, we'll show how lineage integrates with the lakehouse stack alongside tables — providing a natural view of your upstream and downstream data dependencies. Learn practical approaches for implementing lineage using Spark listeners and open standards that work across platforms.
Speaker
Raki Rahman, Principal Software Engineer, SQL Server @ Microsoft
Raki Rahman is a Principal Software Engineer on the SQL Server Telemetry & Intelligence Team - his charter includes operating SQL's large-scale telemetry, agents, ingestion architecture, and data analytics platform spanning millions of database instances inside and outside Azure.
He has built and maintains systems that ingest and processes massive volumes of telemetry using high-throughput, stateful stream processing and multi-petabyte data lakes, including hyperscale control and data-plane service components.
His team specializes in OSS frameworks such as Kubenetes, OpenTelemetry, Apache Spark, Kafka, Grafana and Delta Lake.
Host
Scott Haines, Staff Developer Advocate, Databricks
Scott Haines is a seasoned software engineer specializing in massive distributed data systems and streaming technologies. Over the past 15 years, he’s built and scaled data infrastructure at leading companies, including Yahoo!, Twilio, Nike, and, currently stepped into a role as a Staff Developer Advocate at Databricks.
Scott was a Databricks Beacon/MVP for five years before joining Databricks, he’s the author of books on Apache Spark and Delta Lake, and makes it his mission to help organizations successfully adopt open-source.