DataFlow.Spark 1.1.0

DataFlow.Spark

LINQ-native Apache Spark integration for DataFlow.NET.

📖 Full Documentation

Quality Metrics

Features

• Native LINQ Translation - Write C# LINQ, execute distributed Spark
• Streaming Results - Efficient processing with DataFrames
• Type Safety - Strong typing with automatic column mapping
• Distributed Processing - Scale to petabytes with Apache Spark
• O(1) Memory Writes - Batched streaming for table writes
• Window Functions - Rank, Lead, Lag, running aggregates with expression syntax
• Cases Pattern - Multi-output conditional routing
• In-Memory Push - context.Push(data) for test data injection

Quick Start

using DataFlow.Spark;

// Connect to Spark (local mode)
using var context = Spark.Connect("local[*]", "MyApp");

// Production cluster examples:
// using var context = Spark.Connect("spark://spark-master:7077", "MyApp");
// using var context = Spark.Connect("yarn", "MyApp");

// Query with LINQ (cluster-side execution)
var stats = context.Read.Table<Order>("sales.orders")
    .Where(o => o.Amount > 1000)
    .GroupBy(o => o.Region)
    .Select(g => new { Region = g.Key, Total = g.Sum(o => o.Amount) })
    .ToList();

// Side effects with ForEach (executes on Spark executors - NOT locally!)
context.Read.Table<Order>("sales.orders")
    .ForEach(o => Metrics.Increment("orders_processed"))  // Runs on cluster
    .Show();

Write Operations

// Bulk write with O(1) memory streaming
await data.WriteTable(context, "orders", bufferSize: 10_000).Overwrite();

// File exports with custom buffer size
await data.WriteParquet(context, "hdfs://data/orders.parquet", bufferSize: 50_000);
await data.WriteCsv(context, "s3://bucket/orders.csv", bufferSize: 10_000);

// Async streaming with timeout flush (for IAsyncEnumerable)
await asyncStream.WriteParquet(context, "path.parquet", 
    bufferSize: 5_000, 
    flushInterval: TimeSpan.FromSeconds(30));

// Merge (upsert) into Delta tables
await data.MergeTable(context, "delta/orders", o => o.OrderId, bufferSize: 10_000);

Requirements

• .NET 8.0+
• DataFlow.Net 1.1.0+
• Apache Spark 3.5.0+
• DataFlow.Spark license for production

Support & Issues

📧 Contact: tecnet.paris@gmail.com
🐛 Report Issues: github.com/improveTheWorld/DataFlow.NET/issues

License

Free development tier (DEBUG builds, 1,000 row limit).
Production licensing: Coming soon at https://get-dataflow.net/pricing
📧 Urgent inquiries: tecnet.paris@gmail.com