Pqax.Collector.ParquetNet
0.2.2
Prefix Reserved
dotnet add package Pqax.Collector.ParquetNet --version 0.2.2
NuGet\Install-Package Pqax.Collector.ParquetNet -Version 0.2.2
<PackageReference Include="Pqax.Collector.ParquetNet" Version="0.2.2" />
<PackageVersion Include="Pqax.Collector.ParquetNet" Version="0.2.2" />
<PackageReference Include="Pqax.Collector.ParquetNet" />
paket add Pqax.Collector.ParquetNet --version 0.2.2
#r "nuget: Pqax.Collector.ParquetNet, 0.2.2"
#:package Pqax.Collector.ParquetNet@0.2.2
#addin nuget:?package=Pqax.Collector.ParquetNet&version=0.2.2
#tool nuget:?package=Pqax.Collector.ParquetNet&version=0.2.2
Pqax.Collector.ParquetNet
Reads Delta checkpoints for the PQAX collector. A checkpoint is a Parquet file with nested data, which needs a complete Parquet reader, so the collector takes it through an interface, ICheckpointReader, and this package implements it with Parquet.Net.
| Target | net8.0. Parquet.Net 5.6.1 ships net8.0 and net10.0 assemblies only. |
| Dependencies | Pqax.Collector and Parquet.Net 5.6.1. Version 5.4.0 pulls a Snappier with a known vulnerability, and version 6 changed its API. |
Use
var collector = new PqaxCollector(storage, new CollectorOptions(), new ParquetNetCheckpointReader());
ParquetNetCheckpointReader.ReadAsync(storage, path, cancellation) reads one checkpoint file whole, through the storage of the collection, in one request for a file under 64 MB. It returns its add, remove, metaData, protocol, domainMetadata, sidecar, and checkpointMetadata actions. The collector calls it for a classic checkpoint, for each part of a multi-part checkpoint, for a v2 checkpoint in Parquet, and for each sidecar file.
What it handles
| Case | How |
|---|---|
| A required field inside an optional struct | Parquet.Net returns such a field without the rows where the struct is missing, as the Fabric warehouse writes its checkpoints. The reader lines the values up with the rows from the definition levels. |
Maps: partitionValues, tags, configuration |
Rebuilt row by row from the repetition and definition levels, whatever the writer names the levels between, such as key_value or map |
Lists: partitionColumns, readerFeatures, writerFeatures |
The same, for list/element and other names |
| Statistics | As JSON text in add.stats, which the collector parses, or as add.stats_parsed, from which the reader takes numRecords and the null counts of the top-level columns |
| Deletion vectors | Storage type, location, offset, size, and cardinality: the location and the offset identify the file in the log, and the collector keeps only the size and the cardinality |
A column that does not line up with the rows makes the read fail with an InvalidDataException. The collector turns it into the issue DeltaCheckpointNotRead.
| Product | Versions Compatible and additional computed target framework versions. |
|---|---|
| .NET | net8.0 is compatible. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. net9.0 was computed. net9.0-android was computed. net9.0-browser was computed. net9.0-ios was computed. net9.0-maccatalyst was computed. net9.0-macos was computed. net9.0-tvos was computed. net9.0-windows was computed. net10.0 was computed. net10.0-android was computed. net10.0-browser was computed. net10.0-ios was computed. net10.0-maccatalyst was computed. net10.0-macos was computed. net10.0-tvos was computed. net10.0-windows was computed. |
-
net8.0
- Parquet.Net (>= 5.6.1)
- Pqax.Collector (>= 0.2.2)
NuGet packages
This package is not used by any NuGet packages.
GitHub repositories
This package is not used by any popular GitHub repositories.