DataFlow.Snowflake 1.3.0

DataFlow.Snowflake

LINQ-native Snowflake integration for DataFlow.NET.

dotnet add package DataFlow.Snowflake --version 1.3.0

📖 Full Documentation · 📋 Changelog

Features

• Native LINQ Translation - Write C# LINQ, execute Snowflake SQL
• Streaming Results - Row-by-row processing with IAsyncEnumerable
• Type Safety - Strong typing with automatic column mapping
• SQL Injection Prevention - Parameterized queries by default
• O(1) Memory Writes - Native streaming via PUT + COPY INTO
• Cases Pattern - Multi-output conditional routing
• Auto-UDF — Custom methods in Where/Select/OrderBy/GroupBy auto-translate to Snowflake UDFs (static, instance, lambda, entity-param)
• ForEach — Server-side iteration via stored procedures with static field sync-back
• Pull() Escape Hatch - Switch to client-side streaming for edge cases

Quick Start

using DataFlow.SnowflakeQuery;

// Connect to Snowflake
using var context = Snowflake.Connect(
    account: "xy12345.us-east-1",
    user: "myuser",
    password: "mypass",
    database: "MYDB",
    warehouse: "COMPUTE_WH"
);

// Query with LINQ (server-side SQL)
var orders = await context.Read.Table<Order>("orders")
    .Where(o => o.Amount > 1000)
    .OrderByDescending(o => o.OrderDate)
    .Take(100)
    .ToList();

// Client-side processing requires explicit Pull()
await context.Read.Table<Order>("orders")
    .Where(o => o.Status == "Active")    // Server-side SQL
    .Pull()                               // ← Switch to client
    .ForEach(o => Console.WriteLine(o))   // Client-side C#
    .Do();

Write Operations

Snowflake uses native IAsyncEnumerable streaming - O(1) memory, no config needed:

// Bulk insert (streams via PUT + COPY INTO)
await records.WriteTable(options, "ORDERS");
await records.WriteTable(options, "ORDERS").CreateIfMissing();
await records.WriteTable(options, "ORDERS").Overwrite();

// Upsert (merge) on key
await records.MergeTable(options, "ORDERS", o => o.OrderId);

// Update specific columns only
await records.MergeTable(options, "ORDERS", o => o.OrderId)
    .UpdateOnly("Status", "UpdatedAt");

Nested Objects (VARIANT)

Access Snowflake VARIANT columns with natural C# property syntax:

// Model with nested properties
public class Order {
    public int Id { get; set; }
    
    [Variant]  // Marks column as VARIANT
    public OrderData Data { get; set; }
}

// Query nested properties - translates to colon syntax
var parisOrders = await context.Read.Table<Order>("ORDERS")
    .Where(o => o.Data.Customer.City == "Paris")
    .ToList();
// SQL: WHERE data:customer:city = 'Paris'

Requirements

• .NET 8.0+
• DataFlow.Net 1.2.1+
• DataFlow.Snowflake license for production

Support & Issues

📧 Contact: tecnet.paris@gmail.com
🐛 Report Issues: github.com/improveTheWorld/DataFlow.NET/issues

License

Free Developer Tier (No License Required)

Use DataFlow.Snowflake free for development and testing up to 1,000 rows per query:

Examples:

# Option 1: Set environment variable
$env:DATAFLOW_ENVIRONMENT="Development"  # PowerShell
export DATAFLOW_ENVIRONMENT=Development  # Bash

# Option 2: Launch with debugger attached (auto-detects)
dotnet run --launch-profile "Development"  # Uses launchSettings.json
# Or simply press F5 in Visual Studio/Rider

Production License

For production workloads (unlimited rows), obtain a license at:

• 🌐 Pricing: https://get-dataflow.net/pricing
• 📧 Contact: tecnet.paris@gmail.com

Set your license key as an environment variable (auto-detected at runtime):

# PowerShell
$env:DATAFLOW_LICENSE_KEY="your-license-key"

# Bash/Linux/macOS
export DATAFLOW_LICENSE_KEY="your-license-key"

# Docker / Kubernetes
ENV DATAFLOW_LICENSE_KEY=your-license-key

Security: The license key is never in source code. Set it in your deployment environment (CI/CD secrets, Azure Key Vault, AWS Secrets Manager, etc.)