NVortex 0.1.71-alpha

NetGyre

A high-performance, fully-managed .NET implementation of modern columnar data formats — targeting the Vortex™ format and Apache Parquet™.

Status: early, but usable. There is now a public reading API for Parquet — see Reading a file. Vortex remains internal. What works, validated against reference files:

• Parquet footer and metadata parsing (zero-allocation Thrift compact reader)
• Schema tree reconstruction with Dremel definition/repetition levels
• Page reading: v1 and v2 data pages, dictionary pages
• Encodings: PLAIN, RLE_DICTIONARY, RLE/bit-packed hybrid, DELTA_BINARY_PACKED, DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY, BYTE_STREAM_SPLIT
• Codecs: uncompressed, Snappy, Gzip, Brotli, LZ4 (raw blocks and the deprecated Hadoop framing, detected rather than assumed), and Zstandard — the last of these a from-scratch managed decoder (FSE, Huffman, sequences), so the library still has no native dependency
• Flat columns of every physical type except the deprecated BIT_PACKED level encoding: INT32, INT64, FLOAT, DOUBLE, BOOLEAN, BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY, INT96
• Repeated columns of every supported type, to three levels of nesting: full Dremel assembly to list offsets and validity, telling a null list from an empty one, across the standard three-level LIST, the legacy two-level form, unannotated repeated primitives, and maps

92 columns across the corpus decode identically to Parquet.Net, which is used as a test-only differential oracle.

On the Vortex side: postscript, footer, the dtype tree, the layout tree, segment array trees, and values for the primitive, bool, constant, varbinview and fsst encodings — including nullability, which Vortex expresses as a child array rather than a reserved buffer. Read with hand-written FlatBuffers and protobuf readers: no flatc, no codegen step, no external dependency. Validated against files written by vortex-data 0.84.0.

The segment framing, the FSST layout and the string encodings are undocumented upstream and were recovered from the bytes; they are written up in notes/vortex/reading-the-format.md.

The same column read through both readers agrees: alltypes_plain.parquet and the Vortex file pyarrow converted from it both yield [4, 5, 6, 7, 2, 3, 0, 1].

Both readers work over a batch-oriented I/O layer — in-memory, memory-mapped, RandomAccess, and HTTP range requests with coalescing — so a projection over a wide file is one request rather than one per column.

See notes/ for the research and design record.

Why

• Vortex has no .NET implementation at all — and, more notably, no independent implementation in any language: every non-Rust binding (Java, C/C++, Python) wraps the Rust core. A pure-managed .NET reader/writer would be the first clean-room second implementation of the format.
• Parquet in .NET is served, but not from this premise. The existing managed implementation is good and actively maintained; what is missing is a source-generated, reflection-free, Native-AOT-safe object mapping and a genuinely allocation-free read path. That gap is the protobuf-net argument, applied to columnar data.
• The two formats share their encoding lineage (BtrBlocks / FastLanes / FSST / ALP — ALP is now in both specs), so one codebase can serve both without duplicated work.

See notes/landscape/dotnet-prior-art.md for the survey this is based on.

Documentation

Usage guides live in docs/ and are served by GitHub Pages: reading files, nulls and nesting, the source generator, formats supported, and the performance measurements with their caveats.

Reading a file

using NetGyre.Parquet;
using NetGyre.Columnar;

using ParquetFile file = ParquetFile.Open("trades.parquet");

foreach (ParquetColumn column in file.Columns)
{
    Console.WriteLine(column);   // price (Double), symbol (ByteArray, String), ...
}

// Fixed-width columns come back over a pooled buffer. Disposing returns it.
using PooledColumn<double> price = file.Read<double>(file.GetColumn("price"));
ReadOnlySpan<double> values = price.Data.Span;

// Strings, without allocating one per row unless you ask for it.
BinaryColumnData symbol = file.ReadBinary(file.GetColumn("symbol"));
ReadOnlySpan<byte> utf8 = symbol.GetBytes(0);

Repeated columns assemble into list offsets and validity, and keep the distinction Parquet is careful about — a null list is not an empty one:

NestedColumn<long> tags = file.ReadNested<long>(file.GetColumn("tags.list.item"));

ReadOnlySpan<long> values = tags.Values;                       // hoist; it is a property
ValidityMask leaves = tags.Shape.LeafValidity;

for (int row = 0; row < tags.RowCount; row++)
{
    (int start, int length) = tags.GetRowRange(row);
    bool isNull = tags.Shape.Levels[0].Validity.IsNull(row);   // null list, not empty list

    int index = start;
    foreach (long value in values.Slice(start, length))
    {
        if (!leaves.IsNull(index++))
        {
            Use(value);
        }
    }
}

Opening by path memory-maps rather than reads: the metadata is at the end of the file, and a projection touches only the chunks it needs.

This is the low-level half deliberately — spans, pooling, and the raw list structure. A convenience layer over it is future work, and is being kept separate so that the easy API cannot quietly become the fast one.

Where it stands on performance

Measured against Parquet.Net (the incumbent managed implementation) and ParquetSharp (a C++ binding), one million rows, full job. Full detail and caveats in notes/landscape/measured-comparison.md.

The allocation column is the more interesting one: a scan of a required column moves 18.5 KB where the alternatives move over 15 MB, and triggers no garbage collection at any generation.

Two caveats worth stating rather than burying. NetGyre is doing less work than the alternatives: it hands back physical values, where they map logical types onto CLR types. And against a native floor — the Rust parquet crate — we are ahead on the required and Snappy cases and about 1.65× behind on the optional one, which is the definition-level path and the clearest remaining gap.

The conclusion worth drawing is not "we are fastest" but that the language was never the constraint.

Reading Zstandard

The Zstandard decoder is written from RFC 8878 rather than bound to libzstd, so the library keeps no native dependency. Measured three ways — against native libzstd and against a transliteration of the same C into C# — it is at or ahead of native wherever the work is copying, and about 1.7× behind on entropy-heavy data. The write-up, including why three implementations rather than two, is in notes/design/zstd-managed-vs-native.md.

Principles

• Pure managed. No native assets, no RID-specific packages, no P/Invoke in the shipping path.
• Native AOT and trimming are hard constraints, asserted in the build from day one — which rules out reflection-driven mapping and runtime IL generation.
• Zero allocation in steady-state read loops. Spans, pooled buffers, ref struct readers over borrowed memory.
• Explicit SIMD via System.Runtime.Intrinsics, with scalar fallbacks that are still fast.
• Column-first internals, row-first convenience. The idiomatic POCO API is a thin, source-generated adapter over a public column API — and if it cannot be thin, the column API is wrong.

Repository layout

src/          library projects
tests/        xunit v3, running on Microsoft.Testing.Platform
              TestData/parquet  56 files from apache/parquet-testing
              TestData/vortex   10 files written by vortex-data 0.84.0
benchmarks/   BenchmarkDotNet
notes/        format research, landscape survey, design decisions

The Vortex reference files are generated rather than downloaded — nothing upstream ships .vortex files, and there is no Windows wheel for vortex-data. See tests/NetGyre.Tests/TestData/vortex/generate.py.

Building

Requires the .NET SDK pinned in global.json (10.0.3xx).

dotnet build
dotnet test
dotnet run -c Release --project benchmarks/NetGyre.Benchmarks

Note that dotnet test runs through Microsoft.Testing.Platform (opted into via global.json), which does not accept --nologo — passing it forwards the flag to the test host and yields a confusing "Zero tests ran".

Licence

Apache-2.0.

Vortex™ is a trademark of LF Projects, LLC; Apache Parquet™ and Apache Arrow™ are trademarks of the Apache Software Foundation. This project is not affiliated with, endorsed by, or sponsored by either. See notes/naming-and-trademarks.md.