Parquet.SourceGenerator.Attributes 0.0.3

Parquet.SourceGenerator

A zero-reflection C# Roslyn source generator that emits Parquet serializers and deserializers at compile time, targeting Parquet.Net low-level primitives.

📦 Installation

dotnet add package Parquet.SourceGenerator
dotnet add package Parquet.Net

Parquet.SourceGenerator.Attributes is automatically included as a dependency. Parquet.Net must be referenced directly by your project.

📖 Quick Start Example

using System;
using Parquet.SourceGenerator;

[ParquetSerializable]
public partial record UserEvent
{
    [ParquetColumn("event_id", Order = 1)]
    public Guid Id { get; init; }

    [ParquetColumn("username", Order = 2)]
    public string Username { get; init; } = string.Empty;

    [ParquetColumn("timestamp", Order = 3)]
    [ParquetTimestamp(ParquetTimestampUnit.Microseconds)]
    public DateTime Timestamp { get; init; }

    [ParquetColumn("duration", Order = 4)]
    public TimeSpan Duration { get; init; }
}

Writing Parquet Files

List<UserEvent> events = GetEvents();
using var stream = File.Create("events.parquet");

// Simple write
await events.WriteParquetAsync(stream);

// Chunked row-group streaming write
await events.WriteParquetBatchedAsync(stream, rowGroupSize: 10_000);

// Stream directly from IAsyncEnumerable<T>
IAsyncEnumerable<UserEvent> eventStream = GetAsyncEventStream();
await eventStream.WriteParquetAsync(stream, rowGroupSize: 10_000);

Reading Parquet Files (Sequential & Parallel)

using var stream = File.OpenRead("events.parquet");

// Sequential read
List<UserEvent> events = await UserEventParquetExtensions.ReadParquetAsync(stream);

// Multi-core parallel read across row groups
List<UserEvent> parallelEvents = await UserEventParquetExtensions.ReadParquetParallelAsync(stream, maxDegreeOfParallelism: 4);

// Read from in-memory byte buffer
ReadOnlyMemory<byte> buffer = File.ReadAllBytes("events.parquet");
List<UserEvent> memEvents = await UserEventParquetExtensions.ReadParquetAsync(buffer);

Custom Configuration (ParquetSerializerOptions)

var options = new ParquetSerializerOptions
{
    RowGroupSize = 25_000,
    MaxDegreeOfParallelism = 8,
    CompressionMethod = ParquetCompressionMethod.Zstd
};

await events.WriteParquetBatchedAsync(stream, options: options);

Compression options: None, Snappy (default), Gzip, Lz4, Brotli, Zstd.

⚡ Performance & Benchmarks

Zero-reflection C# source generation vs ParquetSerializer v6 reflection baseline:

📌 Note: BenchmarkDotNet results captured on GitHub Actions. Detailed multi-scale reports (1K, 10K, 100K, 1M rows) are in docs/BENCHMARKS.md.

🌐 Real-World Provenanced Dataset Benchmarks

Fixed public datasets tracked under Git LFS with full cryptographic SHA-256 data provenance:

• TPC-H SF 0.01 LineItem: 60,175 rows, 16 columns (decimals, dates, strings, dictionary encoding)
• Adult Census Income: 32,561 rows, 15 columns (9 categorical dictionary columns)
• Diamonds: 53,940 rows, 10 columns (continuous float metrics & ordinal cuts)

🗜️ TPC-H LineItem Multi-Codec Serialization Throughput (60,175 rows)

🛡️ Compiler Diagnostics

🚀 Native AOT & Cold-Start Performance

Under Native AOT, cold invocations (CLI tools, serverless Lambda functions, batch workers) run 7.2× faster with 72% less memory:

To enable Native AOT:

<PropertyGroup>
    <PublishAot>true</PublishAot>
</PropertyGroup>

🔗 Links & Resources

• GitHub Repository
• Design Documentation
• Contributing Guide
• MIT License