Parquet.SourceGenerator.Attributes 0.0.1-dev.1

<p align="center"> <img src="docs/assets/logo.svg" alt="Parquet.SourceGenerator" width="132" height="132"> </p>

<h1 align="center">Parquet.SourceGenerator</h1>

A zero-reflection C# Roslyn source generator that emits Parquet serializers and deserializers at compile time, targeting Parquet.Net low-level primitives.

⚠️ Status: pre-release, not yet published

This is early work and has not been released. Treat the API as unstable.

• Not on NuGet. There is no published package yet; build from source.
• Not benchmarked. The design avoids reflection and should compare well against expression-tree serialization, but no benchmark results have been published. Run the suite yourself (see Contributing) rather than trusting a number here.
• Native AOT is exercised in CI, with caveats. Every run publishes Parquet.SourceGenerator.AotTest with -r linux-x64, which puts the ILCompiler through the generated code, then executes the resulting native binary — it round-trips a Parquet stream and throws on mismatch. Two limits worth knowing:
  • linux-x64 only. Other runtime identifiers are untested.
  • Parquet.Net itself is not AOT-clean. The publish reports IL2104 (trim warnings) and IL3053 (AOT analysis warnings) against Parquet.dll 6.0.3. The generated code is reflection-free, but it sits on a library that is not, so what this proves is that the paths the test exercises work natively — not that any use of this generator will. Exercise your own schema under AOT before relying on it.

Known limitations

✨ Features

• Zero reflection: schema, serializers and deserializers are emitted at compile time.
• Low-level Parquet.Net primitives: writes via ParquetWriter/row-group writers rather than the reflection-based ParquetSerializer.
• Row-group streaming: WriteParquetBatchedAsync writes in fixed-size row groups, so large sequences never need to be materialized in full.
• IAsyncEnumerable<T> streaming: stream items straight into chunked row groups.
• Parallel reader: ReadParquetParallelAsync distributes object construction across row groups.
• Type support: Guid, DateTime, TimeSpan, Enum, decimal, byte[], string, primitives, and Nullable<T>.
• Compile-time diagnostics: PARQ001–PARQ005 (see below).

📦 Installation

No published package yet. Build from source:

git clone https://github.com/rtkelly13/Parquet.SourceGenerator.git
cd Parquet.SourceGenerator
dotnet build Parquet.SourceGenertor.sln --configuration Release

Once a version is published, two packages are needed:

dotnet add package Parquet.SourceGenerator   # the generator; pulls in the attributes package
dotnet add package Parquet.Net               # the generated code calls into Parquet.Net directly

Parquet.SourceGenerator.Attributes arrives as a dependency of the generator, so it does not need installing separately. Parquet.Net does: the generated readers and writers are built on its low-level primitives, and neither package can declare it for you without pinning you to a version.

📖 Quick Start Example

using System;
using Parquet.SourceGenerator;

[ParquetSerializable]
public partial record UserEvent
{
    [ParquetColumn("event_id", Order = 1)]
    public Guid Id { get; init; }

    [ParquetColumn("username", Order = 2)]
    public string Username { get; init; } = string.Empty;

    [ParquetColumn("timestamp", Order = 3)]
    [ParquetTimestamp(ParquetTimestampUnit.Microseconds)]
    public DateTime Timestamp { get; init; }

    [ParquetColumn("duration", Order = 4)]
    public TimeSpan Duration { get; init; }
}

Writing Parquet Files

List<UserEvent> events = GetEvents();
using var stream = File.Create("events.parquet");

// Simple write
await events.WriteParquetAsync(stream);

// Streaming write in fixed 10,000 row group chunks
await events.WriteParquetBatchedAsync(stream, rowGroupSize: 10_000);

// Stream directly from IAsyncEnumerable<T>
IAsyncEnumerable<UserEvent> eventStream = GetAsyncEventStream();
await eventStream.WriteParquetAsync(stream, rowGroupSize: 10_000);

Reading Parquet Files (Sequential & Multi-Core Parallel)

using var stream = File.OpenRead("events.parquet");

// Sequential read
List<UserEvent> events = await UserEventParquetExtensions.ReadParquetAsync(stream);

// Multi-core parallel read across row groups
List<UserEvent> parallelEvents = await UserEventParquetExtensions.ReadParquetParallelAsync(stream, maxDegreeOfParallelism: 4);

// Read from an in-memory byte buffer
ReadOnlyMemory<byte> buffer = File.ReadAllBytes("events.parquet");
List<UserEvent> memEvents = await UserEventParquetExtensions.ReadParquetAsync(buffer);

Custom Configuration (ParquetSerializerOptions)

var options = new ParquetSerializerOptions
{
    RowGroupSize = 25_000,
    MaxDegreeOfParallelism = 8,
    CompressionMethod = ParquetCompressionMethod.Zstd
};

await events.WriteParquetBatchedAsync(stream, options: options);

CompressionMethod accepts None, Snappy (default), Gzip, Lz4, Brotli and Zstd.

🛡️ Compiler Diagnostics & Guardrails

🤝 Contributing & Community

Contributions are welcome! Please review our community guidelines:

• 📖 Contributing Guide: Build setup, testing, and pull request guidelines.
• 📜 Code of Conduct: Community behavior standards.
• 🛡️ Security Policy: Security vulnerability disclosure process.
• 📝 Changelog: Version history and feature release notes.
• 🏗️ Design documentation: Architecture, attribute API, generator pipeline, and testing strategy.

📄 License

This project is licensed under the MIT License.