Parquet.Net 4.4.6

.NET 6.0 .NET Core 3.1 .NET Standard 2.0
dotnet add package Parquet.Net --version 4.4.6
NuGet\Install-Package Parquet.Net -Version 4.4.6
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Parquet.Net" Version="4.4.6" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add Parquet.Net --version 4.4.6
#r "nuget: Parquet.Net, 4.4.6"
#r directive can be used in F# Interactive, C# scripting and .NET Interactive. Copy this into the interactive tool or source code of the script to reference the package.
// Install Parquet.Net as a Cake Addin
#addin nuget:?package=Parquet.Net&version=4.4.6

// Install Parquet.Net as a Cake Tool
#tool nuget:?package=Parquet.Net&version=4.4.6

Apache Parquet for .NET NuGet Nuget

Icon

Fully portable, managed .NET library to 📖read and ✍️write Apache Parquet files. Targets .NET 7, .NET 6.0, .NET Core 3.1, .NET Standard 2.1 and .NET Standard 2.0.

Runs everywhere .NET runs Linux, MacOS, Windows, iOS, Android, Tizen, Xbox, PS4, Raspberry Pi, Samsung TVs and much more.

Quick Start

Why should I use this? I think you shouldn't. Go away and look at better alternatives, like PyArrow that does it much better in Python. Also I'd rather you use Apache Spark with native support for Parquet and other commercial alternatives. Seriously. Comparing to those, this library is just pure shite, developed in spare time by one person. Despite that, it's a de facto standard for .NET when it comes to reading and writing Parquet files. Why? Because:

  • It has zero dependencies - pure library that just works.
  • It's really fast. Faster than Python and Java implementations.
  • It's .NET native. Designed to utilise .NET and made for .NET developers.

Parquet is designed to handle complex data in bulk. It's column-oriented meaning that data is physically stored in columns rather than rows. This is very important for big data systems if you want to process only a subset of columns - reading just the right columns is extremely efficient.

As a quick start, suppose we have the following data records we'd like to save to parquet:

  1. Timestamp.
  2. Event name.
  3. Meter value.

Or, to translate it to C# terms, this can be expressed as the following class:

class Record {
    public DateTime Timestamp { get; set; }
    public string EventName { get; set; }
    public double MeterValue { get; set; }
}

✍️Writing Data

Let's say you have around a million of events like that to save to a .parquet file. There are three ways to do that with this library, starting from easiest to hardest.

🚤Class Serialisation

The first one is the easiest to work with, and the most straightforward. Let's generate those million fake records:

var data = Enumerable.Range(0, 1_000_000).Select(i => new Record {
    Timestamp = DateTime.UtcNow.AddSeconds(i),
    EventName = i % 2 == 0 ? "on" : "off",
    MeterValue = i 
}).ToList();

Now, to write these to a file in say /mnt/storage/data.parquet you can use the following line of code:

await ParquetConvert.SerializeAsync(data, "/mnt/storage/data.parquet");

That's pretty much it! You can customise many things in addition to the magical magic process, but if you are a really lazy person that will do just fine for today.

🌛Row Based API

Another way to serialise data is to use row-based API. They look at your data as a Table, which consists of a set of Rows. Basically looking at the data backwards from the point of view of how Parquet format sees it. However, that's how most people think about data. This is also useful when converting data from row-based formats to parquet and vice versa. Anyway, use it, I won't judge you (very often).

Let's generate a million of rows for our table, which is slightly more complicated. First, we need to declare table and it's schema:

var table = new Table(
    new DataField<DateTime>("Timestamp"),
    new DataField<string>("EventName"),
    new DataField<double>("MeterName"));

The code above says we are creating a new empty table with 3 fields, identical to example above with class serialisation. We are essentially declaring table's schema here. Parquet format is strongly typed and all the rows will have to have identical amount of values and their types.

Now that empty table is ready, add a million rows to it:

for(int i = 0; i < 1_000_000; i++) {
    table.Add(
        DateTime.UtcNow.AddSeconds(1),
        i % 2 == 0 ? "on" : "off",
        (double)i);
}

The data will be identical to example above. And to write the table to a file:

await table.WriteAsync("/mnt/storage/data.parquet");

Of course this is a trivial example, and you can customise it further.

⚙️Low Level API

And finally, the lowest level API is the third method. This is the most performant, most Parquet-resembling way to work with data, but least intuitive and involves some knowledge of Parquet data structures.

First of all, you need schema. Always. Just like in row-based example, schema can be declared in the following way:

var schema = new ParquetSchema(
    new DataField<DateTime>("Timestamp"),
    new DataField<string>("EventName"),
    new DataField<double>("MeterName"));

Then, data columns need to be prepared for writing. As parquet is column-based format, low level APIs expect that low level column slice of data. I'll just shut up and show you the code:

var column1 = new DataColumn(
    (DataField)schema[0],
    Enumerable.Range(0, 1_000_000).Select(i => DateTime.UtcNow.AddSeconds(i)).ToArray());

var column2 = new DataColumn(
    (DataField)schema[1],
    Enumerable.Range(0, 1_000_000).Select(i => i % 2 == 0 ? "on" : "off").ToArray());

var column3 = new DataColumn(
    (DataField)schema[2],
    Enumerable.Range(0, 1_000_000).Select(i => (double)i).ToArray());

Important thing to note here - columnX variables represent data in an entire column, all the values in that column independently from other columns. Values in other columns have the same order as well. So we have created three columns with data identical to the two examples above.

Time to write it down:

using(Stream fs = System.IO.File.OpenWrite("/mnt/storage/data.parquet")) {
    using(ParquetWriter writer = await ParquetWriter.CreateAsync(schema, fs)) {
        using(ParquetRowGroupWriter groupWriter = writer.CreateRowGroup()) {
            
            await groupWriter.WriteColumnAsync(column1);
            await groupWriter.WriteColumnAsync(column2);
            await groupWriter.WriteColumnAsync(column3);
            
        }
    }
}

What's going on?:?:

  1. We are creating output file stream. You can probably use one of the overloads in the next line though. This will be the receiver of parquet data. The stream needs to be writeable and seekable.
  2. ParquetWriter is low-level class and is a root object to start writing from. It mostly performs coordination, check summing and enveloping of other data.
  3. Row group is like a data partition inside the file. In this example we have just one, but you can create more if there are too many values that are hard to fit in computer memory.
  4. Three calls to row group writer write out the columns. Note that those are performed sequentially, and in the same order as schema defines them.

📖Reading Data

Reading data also has three different approaches, so I'm going to unwrap them here in the same order as above.

🚤Class Serialisation

Provided that you have written the data, or just have some external data with the same structure as above, you can read those by simply doing the following:

Record[] data2 = await ParquetConvert.DeserializeAsync<Record>("/mnt/storage/data.parquet");

This will give us an array with one million class instances similar to this:

Of course class serialisation has more to it, and you can customise it further than that.

🌛Row Based API

A read counterpart to the write example above is also a simple one-liner:

Table tbl = await Table.ReadAsync("/mnt/storage/data.parquet");

This will do the magic behind the scenes, give you table schema and rows, similar to this:

As always, there's more to it.

⚙️Low Level API

And with low level API the reading is even more flexible:

using(Stream fs = System.IO.File.OpenRead("/mnt/storage/data.parquet")) {
    using(ParquetReader reader = await ParquetReader.CreateAsync(fs)) {
        for(int i = 0; i < reader.RowGroupCount; i++) { 
            using(ParquetRowGroupReader rowGroupReader = reader.OpenRowGroupReader(i)) {

                foreach(DataField df in reader.Schema.GetDataFields()) {
                    DataColumn columnData = await rowGroupReader.ReadColumnAsync(df);

                    // do something to the column...
                }
            }
        }
    }
}

This is what's happening:

  1. Create read stream fs.
  2. Create ParquetReader - root class for read operations.
  3. The reader has RowGroupCount property which indicates how many row groups (like partitions) the file contains.
  4. Explicitly open row group for reading.
  5. Read each DataField from the row group, in the same order as it's declared in the schema.

Hint: you can also use web based reader app to test your files, which was created using this library!

Choosing the API

If you have a choice, then the choice is easy - use Low Level API. They are the fastest and the most flexible. But what if you for some reason don't have a choice? Then think about this:

Feature 🚤Class Serialisation 🌛Table API ⚙️Low Level API
Performance high very low very high
Developer Convenience feels like C# (great!) feels like Excel close to Parquet
Row based access easy easy hard
Column based access hard hard easy

Contributing

Any contributions are welcome, in any form. Documentation, code, tests, donations or anything else. I don't like processes so anything goes. If you happen to get interested in parquet development, there are some interesting links.

Product Versions
.NET net5.0 net5.0-windows net6.0 net6.0-android net6.0-ios net6.0-maccatalyst net6.0-macos net6.0-tvos net6.0-windows net7.0 net7.0-android net7.0-ios net7.0-maccatalyst net7.0-macos net7.0-tvos net7.0-windows
.NET Core netcoreapp2.0 netcoreapp2.1 netcoreapp2.2 netcoreapp3.0 netcoreapp3.1
.NET Standard netstandard2.0 netstandard2.1
.NET Framework net461 net462 net463 net47 net471 net472 net48 net481
MonoAndroid monoandroid
MonoMac monomac
MonoTouch monotouch
Tizen tizen40 tizen60
Xamarin.iOS xamarinios
Xamarin.Mac xamarinmac
Xamarin.TVOS xamarintvos
Xamarin.WatchOS xamarinwatchos
Compatible target framework(s)
Additional computed target framework(s)
Learn more about Target Frameworks and .NET Standard.

NuGet packages (20)

Showing the top 5 NuGet packages that depend on Parquet.Net:

Package Downloads
Microsoft.DataPrep The ID prefix of this package has been reserved for one of the owners of this package by NuGet.org.

Microsoft Azure Machine Learning Data Preparation SDK.

ChoETL.Parquet The ID prefix of this package has been reserved for one of the owners of this package by NuGet.org.

Parquet extension to Cinchoo ETL framework

Komodo.Core

Komodo core libraries for crawling (file, object, web, database), parsing (JSON, XML, SQL, Sqlite, HTML, text), postings (inverted index, token extraction), indexing (search), metadata generation, and integrating within your application. Komodo is an information search, metadata, storage, and retrieval platform.

DataFrame.Math

A suite of libraries to provide stats functionality in dotnet for parquet and other formats

Microsoft.ML.Parquet The ID prefix of this package has been reserved for one of the owners of this package by NuGet.org.

ML.NET components for Apache Parquet support.

GitHub repositories (5)

Showing the top 5 popular GitHub repositories that depend on Parquet.Net:

Repository Stars
dotnet/machinelearning
ML.NET is an open source and cross-platform machine learning framework for .NET.
ravendb/ravendb
ACID Document Database
Cinchoo/ChoETL
ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
mukunku/ParquetViewer
Simple windows desktop application for viewing & querying Apache Parquet files
compomics/ThermoRawFileParser
Thermo RAW file parser that runs on Linux/Mac and all other platforms that support Mono
Version Downloads Last updated
4.4.6 590 1/31/2023
4.4.5 319 1/30/2023
4.4.4 309 1/27/2023
4.4.3 244 1/26/2023
4.4.2 86 1/26/2023
4.4.1 308 1/25/2023
4.4.0 356 1/24/2023
4.3.4 242 1/23/2023
4.3.3 479 1/20/2023
4.3.2 207 1/19/2023
4.3.1 113 1/19/2023
4.3.0 502 1/18/2023
4.2.3 607 1/16/2023
4.2.2 865 1/11/2023
4.2.1 958 1/10/2023
4.2.0 117 1/10/2023
4.1.3 13,003 12/21/2022
4.1.2 7,152 12/1/2022
4.1.1 23,100 11/10/2022
4.1.0 27,709 10/13/2022
4.0.2 2,958 10/12/2022
4.0.1 1,326 10/11/2022
4.0.0 14,765 9/22/2022
3.10.0 25,582 9/20/2022
3.9.1 637,500 10/14/2021
3.9.0 210,132 6/25/2021
3.8.6 238,027 3/5/2021
3.8.5 18,041 2/23/2021
3.8.4 136,460 12/13/2020
3.8.3 763 12/10/2020
3.8.2 378 12/10/2020
3.8.1 24,543 11/6/2020
3.8.0 1,989 11/6/2020
3.7.7 240,360 6/25/2020
3.7.6 18,449 6/16/2020
3.7.5 11,144 6/8/2020
3.7.4 86,752 5/19/2020
3.7.2 1,883 5/18/2020
3.7.1 41,271 4/21/2020
3.7.0 24,977 4/19/2020
3.6.0 3,679,228 1/23/2020
3.5.3 10,720 1/8/2020
3.5.2 1,848 1/3/2020
3.5.1 643 12/31/2019
3.5.0 6,259 12/18/2019
3.4.3 5,282 12/16/2019
3.4.2 1,702 12/13/2019
3.4.1 596 12/11/2019
3.4.0 602 12/11/2019
3.3.11 4,855 12/1/2019
3.3.10 22,583 11/6/2019
3.3.9 175,014 8/15/2019
3.3.8 6,898 8/1/2019
3.3.7 549 8/1/2019
3.3.6 607 7/31/2019
3.3.5 22,186 7/5/2019
3.3.4 141,026 3/11/2019
3.3.3 18,346 2/1/2019
3.3.2 22,957 1/21/2019
3.3.1 2,890 1/14/2019
3.3.0 1,646 1/11/2019
3.2.6 1,174 1/11/2019
3.2.5 2,964 1/3/2019
3.2.4 5,890 11/21/2018
3.2.3 21,037 11/7/2018
3.2.2 3,305 10/30/2018
3.2.1 871 10/30/2018
3.2.0 1,457 10/24/2018
3.1.4 928 10/15/2018
3.1.3 715 10/15/2018
3.1.2 22,381 10/11/2018
3.1.1 1,131 10/4/2018
3.1.0 1,003 10/3/2018
3.1.0-preview-390 561 10/3/2018
3.1.0-preview-373 790 10/2/2018
3.0.5 6,947 8/13/2018
3.0.4 926 7/25/2018
3.0.3 790 7/25/2018
3.0.2 1,266 7/24/2018
3.0.1 771 7/24/2018
3.0.0 1,518 7/19/2018
2.1.4 67,576 6/7/2018
2.1.3 245,779 3/30/2018
2.1.2 131,977 1/10/2018
2.1.1 55,966 12/1/2017
2.1.0 1,084 11/29/2017
2.0.1 853 11/27/2017
2.0.0 1,673 11/27/2017
1.5.1 1,585 11/14/2017
1.4.0 4,724 10/23/2017
1.3.0 3,666 9/12/2017
1.2.139 1,918 9/6/2017
1.1.128 1,531 8/15/2017
1.0.114 998 7/31/2017