spark
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
What is spark?
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
Overview of spark
What it is
.NET for Apache Spark is an open-source project that provides high-performance APIs for using Apache Spark from C# and F#. It targets .NET Standard 2.0 and 2.1, and running an application requires a compatible .NET Worker, Apache Spark runtime, and Scala bridge. The Worker runs on Windows, Linux, and macOS using .NET 8, or on Windows using .NET Framework. The latest stable release is 2.4.0, which removes Apache Spark 2.x support and retains the Spark 3.0 through 3.5 runtime lines. Version 2.4.0 also adds Spark 4.0.0–4.0.4 integration on Windows and Linux with .NET 8, the Scala 2.13 bridge, and JDK 17. Deployment guides are available for Azure HDInsight Spark, Amazon EMR Spark, and Databricks on AWS and Azure.
Key features
- Use DataFrames and Spark SQL to work with structured data.
- Use Structured Streaming to process streaming data.
- Supports Apache Spark 3.0 through 3.5 and Spark 4.0.0–4.0.4 (with limitations).
- Provides sample code for DataFrames, Spark SQL, streaming word count, windowed word count, Kafka word count, and TPC-H queries.
- End-to-end test status is tracked for multiple Spark versions on Windows and Linux.
Use cases
.NET for Apache Spark is suitable for big data processing tasks that require the scalability of Apache Spark while using .NET languages. Use cases include working with structured data via DataFrames and Spark SQL, processing streaming data with Structured Streaming, and running benchmark queries such as TPC-H. It can be deployed on Azure HDInsight Spark, Amazon EMR Spark, and Databricks on AWS and Azure.
Getting started
See the repository README for platform-specific instructions for Windows, Ubuntu, and macOS. The getting-started examples use Spark 3.5.3 with Scala 2.12. Install Microsoft.Spark 2.4.0 and use the matching Worker and bridge JAR.
License and pricing
.NET for Apache Spark is licensed under the MIT license. It is open source and free to use.
Based on the project's public README and repository metadata.