Polars.NET.Core 0.2.1-beta1

Polars.NET

High-Performance, DataFrame Engine for .NET, powered by Rust & Apache Arrow.

Supported Platforms: Windows (x64), Linux (x64/ARM64, glibc/musl), macOS (ARM64).

Why Polars.NET exists

This is the game I'd like to play: binding the lightning-fast Polars engine to the .NET ecosystem. And it means a lot of fun to do this.

Installation

C# Users:

dotnet add package Polars.NET 

F# Users:

dotnet add package Polars.FSharp

• Requirements: .NET 8+.
• Hardware: CPU with AVX2 support (x86-64-v3). Roughly Intel Haswell (2013+) or AMD Excavator (2015+). If you have AVX-512 supported CPU, please try to compile Rust core on your machine use RUSTFLAGS='-C target-cpu=native'

Quick Start

C# Example

using Polars.CSharp;
using static Polars.CSharp.Polars; // For Col(), Lit() helpers

// 1. Create a DataFrame
var data = new[] {
    new { Name = "Alice", Age = 25, Dept = "IT" },
    new { Name = "Bob", Age = 30, Dept = "HR" },
    new { Name = "Charlie", Age = 35, Dept = "IT" }
};
var df = DataFrame.From(data);

// 2. Filter & Aggregate
var res = df
    .Filter(Col("Age") > 28)
    .GroupBy("Dept")
    .Agg(
        Col("Age").Mean().Alias("AvgAge"),
        Col("Name").Count().Alias("Count")
    )
    .Sort("AvgAge", descending: true);

// 3. Output
res.Show();
// shape: (2, 3)
// ┌──────┬────────┬───────┐
// │ Dept ┆ AvgAge ┆ Count │
// │ ---  ┆ ---    ┆ ---   │
// │ str  ┆ f64    ┆ u32   │
// ╞══════╪════════╪═══════╡
// │ IT   ┆ 35.0   ┆ 1     │
// │ HR   ┆ 30.0   ┆ 1     │
// └──────┴────────┴───────┘

F# Example

open Polars.FSharp

// 1. Scan CSV (Lazy)
let lf = LazyFrame.ScanCsv "users.csv"

// 2. Transform Pipeline
let res = 
    lf
    |> pl.filterLazy (pl.col "age" .> pl.lit 28)
    |> pl.groupByLazy 
        [ pl.col "dept" ]
        [ 
            pl.col("age").Mean().Alias "AvgAge" 
            pl.col("name").Count().Alias "Count"
        ]
    |> pl.collect
    |> pl.sort ("AvgAge", false)

// 3. Output
res.Show()

Polyglot Notebook Support

Polars.NET works seamlessly with .NET Interactive. Please refer to the example folder for a Polyglot Notebook demo.

Benchmark

• Benchmark Code Here

• Environment:

  • OS: Fedora Linux 43 (KDE Plasma)

  • CPU & Memory: Intel Core i7-10700 @ 2.90GHz，32G RAM

  • Runtime: .NET 10.0.2 (RyuJIT X64), Python 3.14.2

  • Polars.NET & Polars.FSharp Version: v0.2.1-beta1

1. CSV Parsing & GroupBy (100M Rows)

• Winner: Polars.NET

2. Cosine Similarity (100M Elements)

• Winner: TensorPrimitives / AVX2 For pure dense numerical computations, use .NET's TensorPrimitives. Polars.NET incurs overhead for expression planning.

3. Decimal Rolling Average (100M Rows)

• Winner: PLINQ. To prevent OOM exception, DuckDB.NET and Polars.NET are recommended here.

4. NDJSON Parsing (5M Rows)

• Winner: DuckDB DuckDB's projection pushdown on JSON is superior. Polars.NET is still 3x faster than System.Text.Json and 10x faster than Newtonsoft.

5. Excel Parsing (1M Rows, 20 Cols)

• Winner: Polars.NET Leveraging Rust's calamine crate via FFI allows Polars.NET to outperform dedicated .NET Excel libraries.

6. Join (20M Rows)

• Winner: DuckDB / Polars.NET Both utilize high-performance Hash Joins. Traditional C# approaches (LINQ/MDA) are not viable for this scale.

7. Polars.FSharp vs Deedle(Quant, same as 3)

• Winner: Polars.NET 45x faster than Deedle.

8. Polars.NET vs Python brothers(pandas and pypolars)

• UDF

  Winner: Polars.NET. RyuJIT outperforms Python interpreter in UDF case.

• GroupBy (10M Rows)

  Winner: Polars.NET. Very little diff between Polars.NET and pypolars.

Architecture

3-Layer Architecture ensures stability even when the underlying Rust engine changes.

1. Hand-written Rust C ABI layer bridging .NET and Polars. (native_shim)
2. .NET Core layer for dirty works like unsafe ops, wrappers, LibraryImports. (Polars.NET.Core)
3. High level C# and F# API layer here. No unsafe blocks. (Polars.CSharp & Polars.FSharp)

Roadmap & Documentation

• Expanded SQL Support: Full coverage of Polars SQL capabilities (CTEs, Window Functions) to replace in-memory DataTable SQL queries.

• Cloud Support : Direct IO integration with AWS S3, Azure Blob Storage, and Data Lakes.

• Documentation: Docs Here

Contributing

Contributions are welcome! Whether it's adding new expression mappings, improving documentation, or optimizing the FFI layer.

1. Fork the repo.

2. Create your feature branch.

3. Submit a Pull Request.

📄 License

MIT License. See LICENSE for details.