Polars.NET.Core 0.1.0-alpha2

Polars.NET

🚀 High-Performance, AI-Ready DataFrames for .NET, powered by Rust & Apache Arrow.

Polars.NET is not just a binding; it is a production-grade data engineering toolkit for the .NET ecosystem. It brings the lightning-fast performance of the Polars Rust engine to C# and F#, while adding unique, enterprise-ready features missing from official bindings—like seamless Database Streaming, Zero-Copy Interop, and AI-native Vector support.

Why Polars.NET exists

The .NET ecosystem deserves a first-class, production-grade DataFrame engine — not a thin wrapper, not a toy binding, and not a Python dependency in disguise.

Polars.NET is designed for engineers who care about:

• predictable performance

• strong typing

• streaming data at scale

• and long-term system evolution

Why Polars.NET?

1. ⚡ Unmatched Performance

• Rust Core: Built on the blazing fast Polars query engine (written in Rust).

• Lazy Evaluation: Intelligent query optimizer with predicate pushdown, projection pushdown, and parallel execution.

• Zero-Copy: Built on Apache Arrow, enabling zero-copy data transfer between C#, Python, and databases.

2. 🛡️ Enterprise & AI Ready

• Database Streaming (Unique):

  • Read: Stream millions of rows from any IDataReader (SQL Server, Postgres, SQLite) directly into Polars without loading everything into RAM.

  • Write: Stream processed data back to databases via IBulkCopy interfaces using our unique ArrowToDbStream adapter.

3. 🧶 .NET Native Experience

   • Fluent API: Intuitive, LINQ-like API design for C#.

   • Functional API: Idiomatic, pipe-forward (|>) API for F# lovers.

   • Type Safety: Leveraging .NET's strong type system to prevent runtime errors.

   • Feel free to use C#/F# native UDF to integrate with your own logic.

📦 Installation

C# Users:

dotnet add package Polars.NET

F# Users:

dotnet add package Polars.FSharp

🗜 Target Framework

.NET 8 and later

🏁 Quick Start

C# Example

using Polars.CSharp;
using static Polars.CSharp.Polars; // For Col(), Lit() helpers

// 1. Create a DataFrame
var data = new[] {
    new { Name = "Alice", Age = 25, Dept = "IT" },
    new { Name = "Bob", Age = 30, Dept = "HR" },
    new { Name = "Charlie", Age = 35, Dept = "IT" }
};
using var df = DataFrame.From(data);

// 2. Filter & Aggregate
using var res = df
    .Filter(Col("Age") > 28)
    .GroupBy("Dept")
    .Agg(
        Col("Age").Mean().Alias("AvgAge"),
        Col("Name").Count().Alias("Count")
    )
    .Sort("AvgAge", descending: true);

// 3. Output
res.Show();

F# Example

open Polars.FSharp

// 1. Scan CSV (Lazy)
let lf = LazyFrame.ScanCsv "users.csv"

// 2. Transform Pipeline
let res = 
    lf
    |> pl.filterLazy (pl.col "age" .> pl.lit 28)
    |> pl.groupByLazy 
        [ pl.col "dept" ]
        [ 
            pl.col("age").Mean().Alias "AvgAge" 
            pl.col("name").Count().Alias "Count"
        ]
    |> pl.collect
    |> pl.sort ("AvgAge", false)

// 3. Output
res.Show()

🔥 Killer Features (The "Missing" Parts)

1. 🌊 Streaming ETL: Database → Polars → Database

Process millions of rows with constant memory usage using our unique streaming adapters.

// 1. Source: Stream from Database (e.g., SqlDataReader)
// We scan the DB via a factory, pulling 50k rows at a time into Apache Arrow batches.
var lf = LazyFrame.ScanDb(() => mySqlCommand.ExecuteReader(), batchSize: 50_000);

// 2. Transform: Lazy Evaluation (Rust Engine)
// No data is loaded yet. We are building a query plan.
var pipeline = lf
    .Filter(Col("Region") == Lit("US"))
    .WithColumn((Col("Amount") * 1.08).Alias("TaxedAmount"))
    .Select("OrderId", "TaxedAmount", "OrderDate");

// 3. Sink: Stream back to Database (e.g., SqlBulkCopy)
// We expose the processed stream as an IDataReader implementation!
pipeline.SinkTo((IDataReader reader) => 
{
    // This reader pulls data from the Rust engine on-demand.
    // Perfect for SqlBulkCopy.WriteToServer(reader)
    using var bulk = new SqlBulkCopy(connectionString);
    bulk.DestinationTableName = "ProcessedOrders";
    bulk.WriteToServer(reader);
});

┌──────────────┐     Arrow Batches     ┌────────────┐
│  Database    │ ───────────────────▶ │ Polars Core│
│ (IDataReader)│                       │   (Rust)   │
└─────▲────────┘ ◀─────────────────── │            │
      │        Zero-Copy Stream        └─────▲──────┘
      │                                      │
      │                                      │ FFI
┌─────┴──────┐                               │
│   .NET API │ ◀────────────────────────────┘
│ (C# / F#)  │
└────────────┘

2. 🧠 Native C#/F# UDFs

Run C#/F# functions directly on Expr/Series with Zero-Copy overhead using Apache Arrow memory layout.

// Define logic with Option handling (Safe!)
let complexLogic (opt: int option) =
    match opt with
    | Some x when x % 2 = 0 -> Some (x * 10)
    | _ -> None

let s = Series.create("nums", [Some 1; Some 2; None; Some 4])

// Execute MapOption directly on Series
// No need to convert to C# objects or slow IEnumerable!
let result = s.MapOption(complexLogic, DataType.Int32)

// Result: [null, 20, null, 40]

3. 🕒 Time Series Intelligence

Robust support for time-series data, including As-Of Joins and Dynamic Rolling Windows.

// As-Of Join: Match trades to the nearest quote within 2 seconds
var trades = dfTrades.Lazy(); // timestamp, ticker, price
var quotes = dfQuotes.Lazy(); // timestamp, ticker, bid

var enriched = trades.JoinAsOf(
    quotes, 
    leftOn: Col("timestamp"), 
    rightOn: Col("timestamp"),
    by: [Col("ticker")],      // Match on same Ticker
    tolerance: "2s",          // Look back max 2 seconds
    strategy: "backward"      // Find previous quote
);

// F# Dynamic Rolling Window
lf
|> pl.groupByDynamic "time" (TimeSpan.FromHours 1.0)
    [ pl.col("value").Mean().Alias("hourly_mean") ]
|> pl.collect

🗺️ Roadmap & Documentation

We are actively working on detailed API documentation.

- Auto-generated API Reference (HTML)

And plan to migrate core Polars Engine from 0.50 to 0.52(newest)

🤝 Contributing

Contributions are welcome! Whether it's adding new expression mappings, improving documentation, or optimizing the FFI layer.

1. Fork the repo.

2. Create your feature branch.

3. Submit a Pull Request.

📄 License

MIT License. See LICENSE for details.