Polars.NET 1.0.1

<div align="center"> <img src="assets/logo.svg" width="120" height="120" alt="Polars.NET Logo" /> <h1>Polars.NET</h1> <p><b>Blazingly Fast DataFrames & LINQ Engine for .NET</b></p> <p><i>Powered by Rust Polars & Apache Arrow</i></p>

<p> <a href="https://www.nuget.org/packages/Polars.NET"><img src="https://img.shields.io/nuget/v/Polars.NET.svg" alt="NuGet Polars.NET" /></a> <a href="https://www.nuget.org/packages/Polars.NET"><img src="https://img.shields.io/nuget/dt/Polars.NET.svg" alt="NuGet Downloads" /></a> <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg?style=flat-square" alt="License" /></a> <a href="https://errorlsc.github.io/Polars.NET/"><img src="https://img.shields.io/badge/docs-online-512BD4.svg?style=flat-square&logo=read-the-docs&logoColor=white" alt="Documentation" /></a> <a href="https://github.com/sponsors/ErrorLSC"><img src="https://img.shields.io/badge/Sponsor-GitHub-ea4aaa.svg?style=flat-square&logo=githubsponsors&logoColor=white" alt="Sponsor" /></a> </p>

<p> <a href="https://errorlsc.github.io/Polars.NET/"><b>Documentation Website</b></a> | <a href="https://errorlsc.github.io/Polars.NET/articles/quickstart"><b>Quick Start</b></a> | <a href="https://errorlsc.github.io/Polars.NET/articles/linq"><b>LINQ Guide</b></a> | <a href="https://errorlsc.github.io/Polars.NET/api/csharp_generated/DataFrame"><b>C# API</b></a> | <a href="https://errorlsc.github.io/Polars.NET/api/fsharp_generated/pl"><b>F# API</b></a> | <a href="https://errorlsc.github.io/Polars.NET/llms.txt"><b>llms.txt</b></a> | <a href="https://github.com/ErrorLSC/Polars.NET-Cookbook"><b>Cookbook</b></a> </p>

<br /> <img src="assets/comic.jpg" alt="Polars.NET: Rust Polars + .NET LINQ" width="800" /> </div>

I really dreamed that we can play with DataFrames in .NET just like python, now we got a much more enjoyable and funny one.

Highlights

• Native Pushdown LINQ: Compiles C# LINQ and F# query { } directly into Rust Polars physical plans then be optimized & executed natively. Verified with all 22 TPC-H benchmark queries implemented in native LINQ without client-side fallback. Includes .Explain(), .ToDataFrame(), and joins between in-memory collections and DataFrames.
• High Performance: Built on the Rust Polars engine and Apache Arrow columnar memory format. Delivers vectorized SIMD execution, query optimization (predicate and projection pushdown, common subplan elimination), and out-of-core streaming execution for larger-than-memory datasets.
• Deep .NET Ecosystem Integration: First-class bridges to standard .NET abstractions, including IAsyncEnumerable<T> streaming, ADO.NET IDataReader / DbDataReader ingestion, ML.NET IDataView pipelines, and direct extraction into System.Numerics.Tensors (ReadOnlyTensorSpan) for numerical computing and model inference.
• Universal I/O & Storage: Reads and writes Parquet, CSV, JSON/NDJSON, Arrow IPC, Excel, and Avro. Native integration with Arrow Database Connectivity (ADBC) and Delta Lake (ACID transactions, time travel, and SQL MERGE), targeting both local filesystems and cloud object storage (Amazon S3, Azure Blob, Google Cloud Storage).

Installation

Polars.NET targets .NET 8.0 and requires referencing both the managed API package and the matching native runtime package for your execution platform:

C# Users

# C# API package
dotnet add package Polars.NET

# Platform native runtime (choose matching your deployment target):
dotnet add package Polars.NET.Native.win-x64
# dotnet add package Polars.NET.Native.linux-x64
# dotnet add package Polars.NET.Native.linux-arm64
# dotnet add package Polars.NET.Native.osx-arm64

# Optional: ML.NET integration
dotnet add package Polars.NET.ML

F# Users

# F# API package
dotnet add package Polars.FSharp

# Platform native runtime:
dotnet add package Polars.NET.Native.win-x64

Quick Tour

1. C# LINQ Query

Write standard C# LINQ queries then execute them natively without intermediate managed objects:

using Polars.CSharp;
using Polars.CSharp.Linq;

public record Order(int Id, string Customer, string Region, double Revenue);

List<Order> orders = GetOrders();

// 1. Direct entry from in-memory collection, DataFrame, or ADO.NET IDataReader:
var query = from o in orders.AsPolarsQueryable()
            where o.Revenue >= 500.0 && o.Region != "Test"
            group o by o.Region into g
            select new {
                Region = g.Key,
                TotalRevenue = g.Sum(x => x.Revenue),
                OrderCount = g.Count()
            };

// 2. Materialize directly into a Polars DataFrame:
using DataFrame resultDf = query.ToDataFrame();
resultDf.Show();

// 3. Or inspect the execution plan:
query.InspectPlan();

2. C# Fluent API

Polars DSL for complex analytics, column expressions, and Selectors:

using Polars.CSharp;
using Pl = Polars.CSharp.Polars;
using Cs = Polars.CSharp.Polars.Selectors;

// Read Parquet lazily:
using var lf = LazyFrame.ScanParquet("orders/*.parquet");

using var summaryDf = lf
    .Filter(Pl.Col("order_date") >= new DateOnly(2025, 1, 1))
    .GroupBy("region")
    .Agg(
        Pl.Col("revenue").Sum().Alias("total_revenue"),
        Pl.Col("customer_id").NUnique().Alias("unique_customers"),
        Cs.Numeric().Mean().Name.Prefix("avg_")
    )
    .Sort("total_revenue", descending: true)
    .Collect();

summaryDf.Show();

3. F# Query Computation Expression

F# strong typing CE, anonymous records, and pipeline combinators over native queries:

open Polars.FSharp
open Polars.FSharp.Query

// F# query { } compiles into polars execution plan:
let summaryQuery =
    query {
        for o in df.AsQueryable<Order>() do
        where (o.Revenue >= 500.0 && o.Region <> "Test")
        groupValBy o o.Region into g
        select {|
            Region = g.Key
            TotalRevenue = g.Sum(fun x -> x.Revenue)
            OrderCount = g.Count()
        |}
    }

use resultDf = summaryQuery.ToDataFrame()
resultDf |> DataFrame.show

4. Polars.NET Exclusive: Strongly-Typed SQL Merge

Perform ANSI SQL MERGE upsert operations directly on DataFrames or in-memory collections:

List<ProductItem> inMemoryTarget = GetTargetProducts();
List<ProductItem> incomingUpdates = GetSourceProducts();

// Start merge from a List<T>:
using DataFrame mergedDf = inMemoryTarget.Merge(incomingUpdates, t => t.Id, s => s.Id)
    .WhenMatched(
        (t, s) => s.Price > t.Price,
        then => then.Update(set => set.Set(t => t.Price, (t, s) => s.Price))
    )
    .WhenNotMatchedByTarget(then => then.Insert())
    .WhenNotMatchedBySource(t => t.Category == "Seasonal", then => then.Delete())
    .ToDataFrame(); // Or materialize directly to List<ProductItem> via .ToList()

Documentation & Guides

Comprehensive documentation, architecture references, and API catalogs are available on our documentation portal:

Project Architecture

Polars.NET uses a modular three-tier architecture to prevent upstream breaking changes:

+-------------------------------------------------------------------+
|               Polars.NET User API Layer                           |
|   - Polars.CSharp                                                 |
|   - Polars.FSharp                                                 |
+-------------------------------------------------------------------+
                                  |
                                  v
+-------------------------------------------------------------------+
|              Polars.NET.Core & Polars.NET.Linq                    |
|   - Common Infrastructure, LINQ Provider written in F#            |
|   - Apache Arrow C Data Interface                                 |
|   - P/Invoke bindings and unsafe wrappers                         |
+-------------------------------------------------------------------+
                                  |
                                  v
+-------------------------------------------------------------------+
|                 Rust polars & delta-rs Engine                     |
|   - Execution engine                                              |
|   - Physical query planner, optimizer                             |
|   - delta-rs integration                                          |
+-------------------------------------------------------------------+

Benchmarks

vs. Legacy .NET

<div align="center"> <picture> <source media="(prefers-color-scheme: dark)" srcset="assets/benchmark_summary_dark.png"> <img alt="Speedup Summary vs Legacy .NET" src="assets/benchmark_summary_light.png" width="750" /> </picture> </div>

vs. Python Polars

<div align="center"> <picture> <source media="(prefers-color-scheme: dark)" srcset="assets/benchmark_python_dark.png"> <img alt="Performance vs Python Polars" src="assets/benchmark_python_light.png" width="750" /> </picture> </div>

Delta Lake Read & Write vs Python Polars & PySpark

<div align="center"> <img alt="Delta Lake Read Benchmark" src="assets/read_benchmark.png" width="750" /> <br/><br/> <img alt="Delta Lake Write Benchmark" src="assets/write_benchmark.png" width="750" /> </div>

• Benchmark suite: Polars.NET-Benchmark
• Recipe repository: Polars.NET-Cookbook

Sponsoring

If Polars.NET is helpful to you, consider supporting the project via GitHub Sponsors to buy me a cup of black tea. Sláinte!

Acknowledgements

Polars.NET is built on:

• Polars (pola.rs) - The lightning-fast DataFrame library developed by Ritchie Vink and the Polars team.
• Apache Arrow - The columnar in-memory data standard powering zero-copy interop.

License

MIT License. See LICENSE for details.
Third-party licenses and notices are documented in THIRD_PARTY_LICENSES.html.