DataFlow.Spark 1.2.0

DataFlow.Spark

LINQ-native Apache Spark integration for DataFlow.NET.

📖 Full Documentation

Features

• Native LINQ Translation - Write C# LINQ, execute distributed Spark
• Streaming Results - Efficient processing with DataFrames
• Type Safety - Strong typing with automatic column mapping
• Distributed Processing - Scale to petabytes with Apache Spark
• O(1) Memory Writes - Batched streaming for table writes
• Window Functions - Rank, Lead, Lag, running aggregates with expression syntax
• Cases Pattern - Multi-output conditional routing
• In-Memory Push - context.Push(data) for test data injection

Quick Start

using DataFlow.Spark;

// Connect to Spark (local mode)
using var context = Spark.Connect("local[*]", "MyApp");

// Production cluster examples:
// using var context = Spark.Connect("spark://spark-master:7077", "MyApp");
// using var context = Spark.Connect("yarn", "MyApp");

// Query with LINQ (cluster-side execution)
var stats = context.Read.Table<Order>("sales.orders")
    .Where(o => o.Amount > 1000)
    .GroupBy(o => o.Region)
    .Select(g => new { Region = g.Key, Total = g.Sum(o => o.Amount) })
    .ToList();

// Side effects with ForEach (executes on Spark executors - NOT locally!)
context.Read.Table<Order>("sales.orders")
    .ForEach(o => Metrics.Increment("orders_processed"))  // Runs on cluster
    .Show();

Write Operations

// From SparkQuery (server-side, no context needed)
await context.Read.Table<Order>("orders")
    .Where(o => o.Amount > 1000)
    .WriteParquet("/output/high_value");

await context.Read.Table<Order>("orders")
    .WriteTable("analytics.summary").Overwrite();

// From local IEnumerable (client → server, context required)
await data.WriteTable(context, "orders", bufferSize: 10_000).Overwrite();
await data.WriteParquet(context, "hdfs://data/orders.parquet", bufferSize: 50_000);

// From IAsyncEnumerable (streaming client → server)
await asyncStream.WriteParquet(context, "path.parquet", 
    bufferSize: 5_000, 
    flushInterval: TimeSpan.FromSeconds(30));

Test Coverage

Known Issues: 4 expected failures track open bugs:

• BUG-001 (2 tests): Anonymous type materialization - use DTO classes with { get; set; }
• BUG-004 (2 tests): Ternary expressions - use Cases pattern instead

Requirements

• .NET 8.0+
• DataFlow.Net 1.1.0+
• Apache Spark 3.5.0+
• DataFlow.Spark license for production

Support & Issues

📧 Contact: tecnet.paris@gmail.com
🐛 Report Issues: github.com/improveTheWorld/DataFlow.NET/issues

License

Development Tier (Free)

Use DataFlow.Spark free for development and testing up to 1,000 rows per query:

Examples:

# Option 1: Set environment variable
$env:DATAFLOW_ENVIRONMENT="Development"  # PowerShell
export DATAFLOW_ENVIRONMENT=Development  # Bash

# Option 2: Launch with debugger attached (auto-detects)
dotnet run --launch-profile "Development"  # Uses launchSettings.json
# Or simply press F5 in Visual Studio/Rider

Production License

For production workloads (unlimited rows), obtain a license at:

• 🌐 Pricing: https://get-dataflow.net/pricing
• 📧 Contact: tecnet.paris@gmail.com

Set your license key as an environment variable (auto-detected at runtime):

# PowerShell
$env:DATAFLOW_LICENSE_KEY="your-license-key"

# Bash/Linux/macOS
export DATAFLOW_LICENSE_KEY="your-license-key"

# Docker / Kubernetes
ENV DATAFLOW_LICENSE_KEY=your-license-key

Security: The license key is never in source code. Set it in your deployment environment (CI/CD secrets, Azure Key Vault, AWS Secrets Manager, etc.)