Universal.Apache.Parquet
1.0.0
dotnet add package Universal.Apache.Parquet --version 1.0.0
NuGet\Install-Package Universal.Apache.Parquet -Version 1.0.0
<PackageReference Include="Universal.Apache.Parquet" Version="1.0.0" />
<PackageVersion Include="Universal.Apache.Parquet" Version="1.0.0" />
<PackageReference Include="Universal.Apache.Parquet" />
paket add Universal.Apache.Parquet --version 1.0.0
#r "nuget: Universal.Apache.Parquet, 1.0.0"
#:package Universal.Apache.Parquet@1.0.0
#addin nuget:?package=Universal.Apache.Parquet&version=1.0.0
#tool nuget:?package=Universal.Apache.Parquet&version=1.0.0
Universal.Apache.Parquet
Pure managed Parquet reading and writing targeting .NET Standard 2.0, with .NET APIs for schemas, columns, records, exact logical values, indexes, Bloom filters, and modular encryption. Format contracts follow Apache parquet-format revision ec90dd4661ff1c4d77f06b5b5232c5aedf68ffc5; attribution is in NOTICE.
Compression and encryption use managed Universal.Common dependencies. There are no native runtime dependencies.
Quick start
using System.Collections.Generic;
using System.IO;
using Universal.Apache.Parquet;
using Universal.Apache.Parquet.Format;
Schema schema = new Schema(
new Field("id", PhysicalType.Int32) { Repetition = Repetition.Required },
Field.String("name"));
using (FileStream output = File.Create("example.parquet"))
using (ParquetWriter writer = new ParquetWriter(output, schema))
{
writer.WriteRowGroup(
Column.Create(schema.Columns[0], new[] { 1, 2 }),
Column.Create(schema.Columns[1], new string?[] { "Ada", null }));
}
using (FileStream input = File.OpenRead("example.parquet"))
using (ParquetReader reader = new ParquetReader(input))
{
IReadOnlyList<IReadOnlyDictionary<string, object?>> rows = reader.ReadRecords(0);
}
Use Universal.Apache.Parquet and Universal.Apache.Parquet.Format namespaces. The writer requires an empty writable destination. The reader requires a readable seekable source. Streams supplied by the caller remain open. Dispose or complete the writer before consuming its file. ParquetWriter.CreateAsync writes the initial header asynchronously for destinations that forbid synchronous I/O; pair it with async write methods and CompleteAsync before disposal. Instances and their streams must not be used concurrently.
WriteRecords and ReadRecords represent groups as named dictionaries, lists as arrays/sequences, and maps as key/value pairs. Explicit Column values retain physical data and nesting levels. ToLogicalArray restores logical values for non-repeated columns. Exact dates, decimals, times, and timestamps use DateValue, DecimalValue, TimeValue, and TimestampValue; conversions to narrower BCL types reject precision loss. FLOAT16 uses HalfFloat and retains NaN payload bits.
Field.Variant("payload") creates a Variant schema. Write VariantValue.FromObject(value) as its record value; a null record value is SQL null, while VariantValue.FromObject(null) is Variant null. Existing shredded schemas reconstruct and write objects, arrays, and exact primitives, with shared metadata dictionaries. Variant parsing and construction have configurable byte, nesting, dictionary, and value-count limits through ReaderOptions.Variant and WriterOptions.Variant.
FILE groups use FileReference, which validates locator fields and can resolve inline bytes or an external range through a caller-supplied stream resolver. Resolution checks recognized MD5, CRC32, CRC32C, and SHA-256 checksums and leaves supplied streams open. Field.FiniteVector creates canonical fixed-length finite vectors. BSON and geospatial WKB remain binary payloads; JSON and string annotations decode UTF-8. The library does not supply document or GIS query engines.
Unannotated INT96 retains all bits in Int96Value. Its explicit ToTimestamp conversion checks timestamp components rather than assuming every legacy writer produced a valid timestamp.
WriterOptions.ConfigureColumn overrides encoding, compression, dictionary use, page version, and page sizing for each leaf. SortingColumns declares existing sort order; the writer does not sort data. KeyValueMetadata supplies application metadata. Metadata and schema getters return snapshots.
Async row group methods perform asynchronous stream I/O and synchronous encoding/decoding. Writing encodes and writes one page at a time, sharing the same encoding path as synchronous writes. Async reading buffers a compressed column under MaximumBufferedColumnSize. Reader limits control footer, page, column-value, decoded payload, and async buffer sizes. MaximumDecodedColumnBytes defaults to 512 MiB and counts physical value payload bytes, including dictionaries and repeated dictionary values. It is checked before reconstructing delta byte-array prefixes and across pages. It is not a peak process-memory bound: temporary buffers, objects, and nesting levels consume additional memory. The default maximum page size is 256 MiB. Writer page targets default to 1 MiB and dictionaries to 1 MiB; dictionaries exceeding their budget fall back to the configured non-dictionary encoding. MaximumPageValues and MaximumPageValueBytes check repeated rows before allocating page encoding buffers. Binary extrema larger than 4 KiB are omitted rather than inflating page headers. These limits do not bound caller-owned materialized columns or the total retained file metadata. Pages exceeding signed 32-bit sizes cannot currently be materialized.
Plaintext encrypted footers permit projection of unencrypted columns without keys. In that mode their signatures are not authenticated. Supplying decryption options verifies the footer signature and any verification failure is fatal; IsFooterSignatureVerified exposes the result. Encrypted columns always require keys. Reader metadata preserves original annotations; unsupported logical/physical combinations are interpreted as physical values. A recognized modern annotation takes precedence over legacy annotations. ReaderOptions.ValidateFileRowCount optionally checks the declared footer total against row groups; its default permits legacy files whose footer total is incorrect, matching the native reader. Individual column and row-group counts remain validated. Writers normalize compatible legacy/modern annotations and reject invalid logical values before writing a row group.
Supported features
| Area | Current behavior |
|---|---|
| Physical types | BOOLEAN, INT32, INT64, INT96, FLOAT, DOUBLE, BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY |
| Encodings | PLAIN, dictionary, hybrid RLE/bit packing, legacy bit-packed levels, delta binary/length/byte arrays, byte stream split, ALP |
| Pages | V1/V2, dictionary pages, multiple pages and row groups, CRC32 |
| Compression | Uncompressed, Snappy, GZIP, Brotli, LZ4 raw, legacy Hadoop LZ4, Zstandard, raw LZO1X |
| Nesting | Definition/repetition levels, groups, standard and legacy lists, maps, record reconstruction |
| Logical values | Exact decimals, dates, times, timestamps including the 96-bit carrier, UUID, FLOAT16, intervals, signed/unsigned integers, strings, JSON, BSON and WKB payloads |
| Complex logical types | Variant binary format and shredding, FILE references, finite vectors |
| Metadata | Generated contracts for the pinned specification, unknown-field skipping, application metadata, sort declarations, size histograms, encoding counts, and producer-supplied geospatial statistics |
| Statistics | Column/page extrema and null counts, numeric unsigned ordering, decimal ordering, IEEE total ordering, chronological INT96 ordering, NaN counts, column and offset indexes |
| Bloom filters | Split-block filters with XXH64, including encrypted filters |
| Encryption | AES-GCM and AES-GCM-CTR, encrypted/plaintext signed footers, per-column keys, stored and externally supplied AAD prefixes, encrypted indexes |
Practical limits
Predicate evaluation and automatic page pruning are application responsibilities; callers can inspect indexes, statistics, and Bloom filters. External-column resolution is an explicit extension and is separate from FILE logical values. Zstandard frames requiring an external compression dictionary are rejected because Parquet supplies no such dictionary. Unknown future encodings, codecs, and Variant versions fail explicitly.
Materialized pages and columns use signed 32-bit .NET arrays and configurable resource limits. Pages exceeding configured limits or the supported .NET array size are rejected.
Original Universal.Apache.Parquet work is dedicated under CC0-1.0. The Apache-derived format contracts in Universal.Apache.Parquet/Format.g.cs remain Apache-2.0, with attribution in NOTICE. The package license expression is CC0-1.0 AND Apache-2.0, reflecting separate components; it does not offer a choice of licenses for Apache-derived material. LICENSE defines the scope, and both complete license texts are included as LICENSE.CC0-1.0.txt and LICENSE.Apache-2.0.txt. Apache test fixtures and reference values derived from them retain their upstream licensing; fixture attribution is under TestData/Apache. The managed Brotli codec uses canonical dictionary and transform data under the MIT license, with its license and reproducible provenance included in the compression project and package.
| Product | Versions Compatible and additional computed target framework versions. |
|---|---|
| .NET | net5.0 was computed. net5.0-windows was computed. net6.0 was computed. net6.0-android was computed. net6.0-ios was computed. net6.0-maccatalyst was computed. net6.0-macos was computed. net6.0-tvos was computed. net6.0-windows was computed. net7.0 was computed. net7.0-android was computed. net7.0-ios was computed. net7.0-maccatalyst was computed. net7.0-macos was computed. net7.0-tvos was computed. net7.0-windows was computed. net8.0 was computed. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. net9.0 was computed. net9.0-android was computed. net9.0-browser was computed. net9.0-ios was computed. net9.0-maccatalyst was computed. net9.0-macos was computed. net9.0-tvos was computed. net9.0-windows was computed. net10.0 was computed. net10.0-android was computed. net10.0-browser was computed. net10.0-ios was computed. net10.0-maccatalyst was computed. net10.0-macos was computed. net10.0-tvos was computed. net10.0-windows was computed. |
| .NET Core | netcoreapp2.0 was computed. netcoreapp2.1 was computed. netcoreapp2.2 was computed. netcoreapp3.0 was computed. netcoreapp3.1 was computed. |
| .NET Standard | netstandard2.0 is compatible. netstandard2.1 was computed. |
| .NET Framework | net461 was computed. net462 was computed. net463 was computed. net47 was computed. net471 was computed. net472 was computed. net48 was computed. net481 was computed. |
| MonoAndroid | monoandroid was computed. |
| MonoMac | monomac was computed. |
| MonoTouch | monotouch was computed. |
| Tizen | tizen40 was computed. tizen60 was computed. |
| Xamarin.iOS | xamarinios was computed. |
| Xamarin.Mac | xamarinmac was computed. |
| Xamarin.TVOS | xamarintvos was computed. |
| Xamarin.WatchOS | xamarinwatchos was computed. |
-
.NETStandard 2.0
- System.IO.Hashing (>= 11.0.0)
- Universal.Common.IO.Compression (>= 1.4.0)
- Universal.Common.Security.Cryptography (>= 1.1.0.1)
NuGet packages
This package is not used by any NuGet packages.
GitHub repositories
This package is not used by any popular GitHub repositories.
| Version | Downloads | Last Updated |
|---|---|---|
| 1.0.0 | 83 | 11/12/2026 |
Initial managed Parquet reader and writer with nested records, logical types, all specified codecs, indexes, Bloom filters, and modular encryption.