Spark Connect Meetup
This online webinar is a kick-off to the regular cadence of the Bay Area Apache Spark™ Meetup and webinar series for 2025, where we will share with the community all things Apache Spark and beyond.
We want to share technical presentations on Spark Connect and its ecosystem and integrations for this virtual webinar. Spark Connect enables you to access Apache Spark from anywhere, including from non-JVM language clients, using your choice of IDE, notebook, local laptop, and REPL. In particular, you will learn:
What is Spark Connect, how does it works, why is it essential, and what is its ecosystem
How to access Apache Spark from anywhere with a Python client
Using Spark Connect with non-JVM clients such as Rust and .NET
How you can contribute to Spark Connect
Please join us and learn more about Spark Connect 🤝
Agenda:
Talk 1: Acces Apache Spark™ from anywhere: A Spark client in Python powered by Spark Connect by Martin Grund
Talk 2: Leveraging Apache Spark™ with Rust by Steven Russo
Talk 3: Spark and .NET, from peering in from the cold to joining the party by Ed Elliott
Talk Descriptions:
Talk 1: Use Apache Spark™ from anywhere: A Spark client in Python powered by Spark Connect
Abstract: Spark Connect introduces a client-server architecture for Apache Spark, enabling remote connectivity to Spark clusters via the DataFrame API. This separation allows Spark to be integrated into various environments, including data applications, IDEs, and notebooks. The talk demonstrates how easily Spark Connect can connect to Spark from different platforms.
Bio: Martin Grund is a Principal Software Engineer and Tech Lead at Databricks, working at the intersection between query processing, data governance, and security. He's currently leading the design and engineering efforts for Spark Connect as part of Apache Spark™. Martin has previously led the engineering for Amazon Redshift Spectrum and worked on Cloudera Impala. He holds a PhD in computer science from the Hasso-Plattner Institute in Germany.
Talk 2: Leveraging Apache Spark™ with Rust
Abstract: Spark Connect is extensible, allowing you to write clients in non-JVM languages, such as Rust. As such, you can now write Spark applications in Rust or any other client language. The Rust client from spark-connect-rs interacts with Apache Spark™ via gRPC, sending logical plans to the Spark server and receiving results as Apache Arrow Record Batches. We’ll walk through how to use the spark-connect-rs library to connect to Spark clusters, perform complex data transformations, and execute distributed tasks—all from the safety and speed of Rust.
Bio: Steve Russo is the Lead Data Engineer at a Fortune 50 company, and OSS contributor, and an avid Apache Spark user.
Talk 3: Spark and .NET, from peering in from the cold to joining the party
Abstract: A brief history of using Apache Spark from .NET to explain the current options for using Spark from your favorite language. We will look at how we can use the Spark Connect API, whether by making your own gRPC calls or using a library that gives you a DataFrame API.
Bio: Ed is a data engineer with a background in software engineering who could not be more excited that the Spark Connect API allows us to use the power of Apache Spark from our preferred languages.