Polars.NET.Core 1.0.0

<div align="center"> <img src="assets/logo.svg" width="160" height="160" alt="Polars.NET Logo" /> <h1>Polars.NET</h1> <p><b>Blazingly Fast, Vectorized DataFrames & LINQ Engine for .NET</b></p> <p><i>Powered by Rust Polars & Apache Arrow with zero-copy interoperability, native expression compilation, and full cloud lakehouse support.</i></p>

<p> <a href="https://www.nuget.org/packages/Polars.NET"><img src="https://img.shields.io/nuget/v/Polars.NET.svg" alt="NuGet Polars.NET" /></a> <a href="https://www.nuget.org/packages/Polars.FSharp"><img src="https://img.shields.io/nuget/v/Polars.FSharp.svg" alt="NuGet Polars.FSharp" /></a> <a href="https://www.nuget.org/packages/Polars.NET"><img src="https://img.shields.io/nuget/dt/Polars.NET.svg" alt="NuGet Downloads" /></a> <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg?style=flat-square" alt="License" /></a> <a href="https://errorlsc.github.io/Polars.NET/"><img src="https://img.shields.io/badge/docs-online-512BD4.svg?style=flat-square&logo=read-the-docs&logoColor=white" alt="Documentation" /></a> </p>

<p> <a href="https://errorlsc.github.io/Polars.NET/"><b>Documentation Website</b></a> | <a href="https://errorlsc.github.io/Polars.NET/articles/quickstart"><b>Quick Start</b></a> | <a href="https://errorlsc.github.io/Polars.NET/articles/linq"><b>LINQ Guide</b></a> | <a href="https://errorlsc.github.io/Polars.NET/api/csharp_generated/DataFrame"><b>C# API</b></a> | <a href="https://errorlsc.github.io/Polars.NET/api/fsharp_generated/pl"><b>F# API</b></a> | <a href="https://errorlsc.github.io/Polars.NET/llms.txt"><b>llms.txt</b></a> | <a href="https://github.com/ErrorLSC/Polars.NET-Cookbook"><b>Cookbook</b></a> </p> </div>

Highlights

• 100% Native Pushdown LINQ: Translates C# IQueryable<T> and F# query { } computation expressions directly into native Rust LazyFrame physical plans. Zero intermediate .NET row object allocations on fully pushed down queries. Passed all TPC-H queries test for both C# and F#.
• Blazing Fast SIMD & Multithreading: Vectorized query execution backed by the Rust Polars engine, delivering 10x-50x speedups over legacy .NET data processing frameworks.
• Truly Idiomatic .NET Experience: Ingest collections directly via DataFrame.FromRows(list), pipe database streams using reader.AsPolarsQueryable<T>(), perform ANSI SQL Merge on List<T>, and enumerate rows via zero-GC stack-allocated ref struct cursors.
• Seamless In-Memory Interop: LINQ queries can directly .Join(), .Concat(), or .Union() an arbitrary in-memory List<T> against a high-volume DataFrame query with automatic native Arrow transposition.
• Enterprise Lakehouse & Arrow: Zero-copy Apache Arrow C Data Interface, streaming ADO.NET IDataReader, Delta Lake ACID transactions with full time-travel, and cloud object store integration (Amazon S3, Azure Blob, Google Cloud Storage).

Installation

Polars.NET targets .NET 8.0+ and requires referencing both the managed API package and the matching native runtime package for your execution platform:

C# Users

# Core API package
dotnet add package Polars.NET

# Platform native runtime (choose matching your deployment target):
dotnet add package Polars.NET.Native.win-x64
# dotnet add package Polars.NET.Native.linux-x64
# dotnet add package Polars.NET.Native.linux-arm64
# dotnet add package Polars.NET.Native.osx-arm64

# Optional: LINQ expression tree compiler
dotnet add package Polars.NET.Linq

# Optional: ML.NET integration
dotnet add package Polars.NET.ML

F# Users

# F# idiomatic API package
dotnet add package Polars.FSharp

# Platform native runtime:
dotnet add package Polars.NET.Native.win-x64

# Optional: F# Query CE & LINQ integration
dotnet add package Polars.NET.Linq

Quick Tour

1. C# LINQ Query (IQueryable<T>)

Write standard C# LINQ queries that compile directly into Rust SIMD kernels without hydrating managed objects:

using Polars;
using Polars.CSharp;
using Polars.CSharp.Linq;

public record Order(int Id, string Customer, string Region, double Revenue);

List<Order> orders = GetOrders();

// 1. Direct entry from in-memory collection, DataFrame, or ADO.NET IDataReader:
var query = from o in orders.AsPolarsQueryable()
            where o.Revenue >= 500.0 && o.Region != "Test"
            group o by o.Region into g
            select new {
                Region = g.Key,
                TotalRevenue = g.Sum(x => x.Revenue),
                OrderCount = g.Count()
            };

// 2. Materialize directly into a Polars DataFrame (Zero heap allocations):
using DataFrame resultDf = query.ToDataFrame();
resultDf.Show();

// 3. Or inspect the compiled native plan:
Console.WriteLine(query.Explain());

2. C# Vectorized Expressions (LazyFrame & Pl.Col)

Harness the full power of the Polars DSL for complex analytics, column expressions, and Selectors:

using Polars.CSharp;
using Pl = Polars.CSharp.Polars;
using Cs = Polars.CSharp.Polars.Selectors;

// Read high-throughput Parquet lazily:
using var lf = LazyFrame.ScanParquet("orders/*.parquet");

using var summaryDf = lf
    .Filter(Pl.Col("order_date") >= new DateOnly(2025, 1, 1))
    .GroupBy("region")
    .Agg(
        Pl.Col("revenue").Sum().Alias("total_revenue"),
        Pl.Col("customer_id").NUnique().Alias("unique_customers"),
        Cs.Numeric().Mean().Prefix("avg_")
    )
    .Sort("total_revenue", descending: true)
    .Collect();

summaryDf.Show();

3. F# Query Computation Expression

Leverage F# strong typing, anonymous records, and pipeline combinators over native queries:

open Polars.FSharp
open Polars.FSharp.Query

// F# query { } compiles into native Rust DAG:
let summaryQuery =
    query {
        for o in df.AsQueryable<Order>() do
        where (o.Revenue >= 500.0 && o.Region <> "Test")
        groupValBy o o.Region into g
        select {|
            Region = g.Key
            TotalRevenue = g.Sum(fun x -> x.Revenue)
            OrderCount = g.Count()
        |}
    }

use resultDf = summaryQuery.ToDataFrame()
resultDf.Show()

4. Polars.NET Exclusive: Strongly-Typed SQL Merge

Perform complete ANSI SQL MERGE upsert operations directly on DataFrames or in-memory collections:

List<ProductItem> inMemoryTarget = GetTargetProducts();
List<ProductItem> incomingUpdates = GetSourceProducts();

// Start merge directly from an in-memory List<T>:
using DataFrame mergedDf = inMemoryTarget.Merge(incomingUpdates, t => t.Id, s => s.Id)
    .WhenMatched(
        (t, s) => s.Price > t.Price,
        then => then.Update(set => set.Set(t => t.Price, (t, s) => s.Price))
    )
    .WhenNotMatchedByTarget(then => then.Insert())
    .WhenNotMatchedBySource(t => t.Category == "Seasonal", then => then.Delete())
    .ToDataFrame(); // Or materialize directly to List<ProductItem> via .ToList()

Documentation & Guides

Comprehensive documentation, architecture references, and API catalogs are available on our documentation portal:

Architecture at a Glance

Polars.NET uses a modular three-tier architecture to deliver native speed without sacrificing .NET safety:

+-------------------------------------------------------------------+
|                     User Application Layer                        |
|   C# LINQ (IQueryable<T>)  |  C# Fluent DSL  |  F# Query CE / FP  |
+-------------------------------------------------------------------+
                                  |
                                  v
+-------------------------------------------------------------------+
|               Polars.NET High-Level Managed Layer                 |
|   - Polars.CSharp (DataFrame, LazyFrame, Series, Expr, Selectors) |
|   - Polars.FSharp (Idiomatic modules, struct enumerators)         |
|   - Polars.NET.Linq (Expression tree compiler & plan generator)   |
+-------------------------------------------------------------------+
                                  |
                                  v
+-------------------------------------------------------------------+
|              Polars.NET.Core (FFI & Interop Layer)                |
|   - SafeHandle lifetime management (Zero-leak guarantee)          |
|   - Apache Arrow C Data Interface zero-copy bridge                |
|   - P/Invoke bindings & NativeShim library loaders                |
+-------------------------------------------------------------------+
                                  |
                                  v
+-------------------------------------------------------------------+
|                 Rust Polars Native Engine                         |
|   - Multithreaded chunked execution engine                        |
|   - SIMD vectorization & memory-mapped I/O                        |
|   - Physical query planner, optimizer & streaming sink            |
+-------------------------------------------------------------------+

Benchmarks

Polars.NET brings the blazing performance of Rust Polars directly to the .NET ecosystem:

<div align="center"> <picture> <source media="(prefers-color-scheme: dark)" srcset="assets/benchmark_summary_dark.png"> <img alt="Speedup Summary vs Legacy .NET" src="assets/benchmark_summary_light.png" width="750" /> </picture> </div>

• Reproducible benchmark suite: Polars.NET-Benchmark
• End-to-end recipe repository: Polars.NET-Cookbook

Acknowledgements

Polars.NET stands on the shoulders of giants:

• Polars (pola.rs) - The lightning-fast DataFrame library developed by Ritchie Vink and the Polars team.
• Apache Arrow - The columnar in-memory data standard powering zero-copy interop.

License

MIT License. See LICENSE for details.
Third-party licenses and notices are documented in THIRD_PARTY_LICENSES.html.