Vorticity.RowEncoding 0.7.1

Vorticity

Vorticity reads, writes and queries columnar files in .NET. It speaks Vortex and Apache Parquet, works on a local disk or straight against S3, Azure Blob Storage and Google Cloud Storage, and is written in C# from end to end, with no native library and no third-party dependency.

It is also built for speed: it reads and writes Vortex faster than Vortex's own Rust implementation, its group bys beat DuckDB's on every query we measured, and the managed Zstandard it relies on decompresses faster than the native libzstd. The figures are at the end of this page.

Quick start

The packages target .NET 11. The generator runs at build time only, and PrivateAssets keeps it out of whatever your project packs:

<PackageReference Include="Vorticity" Version="0.6.0" />
<PackageReference Include="Vorticity.Generators" Version="0.6.0" PrivateAssets="all" />

A record declares the columns of a file. It is a schema rather than a row: each member is a column, in declaration order and under its own name.

using Vorticity;

[VortexRecord]
public partial record struct Reading(int Day, double? Celsius, string City);

Write the records, then ask the file a question:

await using (VortexFileWriter writer = VortexSession.Default.CreateWriter<Reading>("readings.vortex"))
{
    await writer.WriteAsync<Reading>(readings);   // a Reading[], copied in column by column
    await writer.CompleteAsync();                 // disposed without it, the file is deleted
}

await using VortexFile file = await VortexFile.OpenAsync("readings.vortex");

long hotDays = await file.Scan<Reading>()
    .Where(r => r.Day >= 900 && r.City == "Paris" && r.Celsius > 30.0)
    .CountAsync();

The lambda given to Where runs once, when the scan is built, and what it records becomes the plan. The file's statistics, zone maps and indexes then skip whatever they can before a block is decoded, and the count comes from the statistics when they settle it, from the encoded blocks otherwise, never from materialised rows. A filter the scan cannot push down, such as r.Day % 2 == 0, does not compile.

A group by is a query too, written as a method chain or in LINQ's query syntax:

var hottest =
    from r in file.Scan<Reading>()
    where r.Day >= 900
    group r by r.City into g
    orderby g.Average(x => x.Celsius) descending
    select (g.Key, g.Count(), g.Average(x => x.Celsius));

await foreach (CityHeat city in hottest.Take(3).As<CityHeat>().ToRecordsAsync())
{
    Console.WriteLine($"{city.City}: {city.Mean:F1} °C over {city.Readings} readings");
}

[VortexRecord]
public partial record struct CityHeat(string City, long Readings, double? Mean);

When the values themselves are what you need, a scan lends each batch's columns as spans over the decoded buffers, with no copy and next to no allocation per batch:

long hot = 0;
await foreach (var (_, celsius, _) in file.Scan<Reading>().Where(r => r.City == "Paris"))
{
    ReadOnlySpan<double> values = celsius.Values;   // the decoded buffer, valid until the next batch
    for (int i = 0; i < values.Length; i++)
    {
        if (celsius.IsValid(i) && values[i] > 30.0) hot++;
    }
}

ToRecordsAsync() yields one record per row when rows are what you need, and a file whose schema is only known at run time is read by column name, as in file.Scan("Day", "City").Where($"Day >= {min}").

Parquet

A Parquet file goes through the same engine, so the same records, filters, aggregates and group bys work on it, and one scan converts it to Vortex:

using Vorticity.Parquet;

#pragma warning disable VX0003   // the Parquet API is experimental until its surface settles

await using ParquetFile parquet = await ParquetFile.OpenAsync("readings.parquet");
double? lyon = await parquet.Scan<Reading>().Where(r => r.City == "Lyon").AverageAsync(r => r.Celsius);

await using VortexFileWriter writer = VortexSession.Default.CreateWriter<Reading>("readings.vortex");
await writer.WriteAsync(parquet.Scan<Reading>());
await writer.CompleteAsync();

VortexSession.Default.CreateParquetWriter<Reading>(path) writes Parquet the same way. Row group statistics, the page index, Bloom filters and dictionaries prune a scan before a page is read.

Object storage

With Vorticity.ObjectStorage, a location opens and writes like a path:

using Vorticity.ObjectStorage;

await using VortexSession session = VortexSession.Create(o => o.ObjectStores.UseEnvironment());

await using VortexFile file = await session.OpenAsync("s3://lake/readings/2026-10.vortex");
double? mean = await file.Scan<Reading>().Where(r => r.Day >= 900).AverageAsync(r => r.Celsius);

UseEnvironment serves every s3://, az:// and gs:// location from the environment, which it reads the way each cloud's own SDKs do, Kubernetes workload identities included. AWS_ENDPOINT_URL_S3 points s3:// at R2, OVHcloud, Scaleway or any other service that speaks the S3 API. Opening a file costs a single request on Amazon S3, a scan coalesces each batch's reads into ranges of up to 16 MiB issued together, and a large write uploads its parts in parallel as they fill. Parquet files open from a local path for now. object-storage.md covers stores, credentials, retries and what each operation costs.

Building from source

global.json pins a .NET 11 release candidate SDK, so until .NET 11 ships, a machine with only .NET 10 is refused before anything builds. That SDK is all dotnet build and dotnet test need. A Rust toolchain is needed only to compare against the Rust reference: the cross-check, the bench's ratio gates and the corpus generator. CONTRIBUTING.md says what to run before a change is done.

Documentation

Performance

Where Vorticity is compared with another implementation, the figure is a speedup: the other side's time divided by Vorticity's, so anything above 1.00x means Vorticity took less time. The machines are an Apple M4 Pro (arm64, 14 cores, macOS) and an AMD Ryzen 9 7950X (x64, 16 cores and 32 threads, Windows 11). The figures against Rust were measured on 2026-10-14 on the M4 Pro and on 2026-10-12 on the Ryzen, those for Parquet on 2026-10-12 and those against DuckDB on 2026-10-14, and how it is measured says what both sides do.

Against Vortex's Rust implementation

Vortex Rust 0.86.1 and Vorticity, each in its own process, read and write a table of four columns and a million rows. On the file Vorticity writes:

Rust keeps the lead in a few places on the M4 Pro: a prefix filter over FSST strings (0.94x) and 10 of the 61 encodings decoded alone, from 0.86x to 1.00x, mostly columns stored plain or masked. The Pcodec files decode in less time there, from 1.07x to 3.59x. On the Ryzen, Vorticity decodes all 61 encodings in less time, Pcodec files included.

Against DuckDB

DuckDB 1.5.2 answers the same group bys on the same files with the same number of threads, from a table of its own loaded beforehand and through its Vortex extension. Vorticity leads all 13 queries on both, from 40 million rows grouped by 10³ to 10⁷ keys to db-benchmark's group by on 10 million rows:

The narrowest lead is db-benchmark's q10 on 14 threads, six keys and nearly a group per row, at 1.18x. These figures come from the M4 Pro.

Parquet

Parquet reads through the same engine as Vortex: a page decompresses straight into the buffer that becomes its column, plain values and dictionaries are read as views rather than copied, and a scan of the taxi trips below, 3 million rows, allocates 82 KiB. The table times a scan of every column of three real files on the Ryzen 9 7950X, against the same rows rewritten as Vortex:

On one thread the codec weighs most: inflating the taxi trips' GZIP pages takes 121 ms of the 158, even though Vorticity decodes GZIP with its own DEFLATE decoder, 19 % faster than .NET's GZipStream.

Zstandard and LZ4

The codecs come from ManagedCompressions, which writes the exact bytes of libzstd 1.5.7 and lz4 1.10.0. Against the native libraries, on silesia.tar in one call, at the levels in use:

On x64, native libzstd decodes with hand-written assembly that arm64 does not have, which is why the managed decoder's lead is smaller there.

How it is measured

Against Rust, both sides are compiled ahead of time, run on the same machine and do the same work. Each opens the file anew at every call, decodes every value it returns to its plain form and must return the same rows, and a write hands its bytes to a sink that keeps none. On the M4 Pro both read the file mapped, where it lies. Under Windows, Rust maps the file while Vorticity copies the segments it reads, as a session that keeps no mapping does there. On both machines both sides also read every value they return, the same bytes, as the Parquet figures read theirs on the caller's thread. Rust is built the way upstream builds its own benchmarks, and on one core its figure is the faster of its two ways of splitting a scan. Against DuckDB, the two sides take turns in pairs and must give the same answer to every query. The page cache is warm, cores are not pinned, and your data and your machine will differ.

Every figure is on the arm64 page, the x64 page, the DuckDB page and the Parquet page, each with its machine, commit and date. 05-benchmarks.md explains what is compared and how, and bench/gate.sh runs everything that gates a change before it is pushed.

Status and license

Vorticity is developed with the help of LLMs, and its API may still change in breaking ways before v1.0. It is licensed under Apache-2.0, as LICENSE and NOTICE say.

Vortex™ is a trademark of LF Projects, LLC, DuckDB is a trademark of the DuckDB Foundation, and Apache Parquet is a trademark of The Apache Software Foundation. Vorticity is an independent implementation, not affiliated with or endorsed by any of them.