FlatFlow.SourceGenerators 1.0.1

FlatFlow

A .NET library for reading and writing fixed-length and delimited flat files. FlatFlow combines span-based zero-allocation parsing with source-generated, AOT-safe mappers and exposes three independently usable layers — so you tap exactly as much of the stack as you need.

Installation

Install the packages you need via the .NET CLI or NuGet Package Manager.

Core runtime (required):

dotnet add package FlatFlow

Attribute definitions (if you want to reference attributes in a separate project):

dotnet add package FlatFlow.Attributes

Source generators (compile-time mappers — no reflection, AOT-safe):

dotnet add package FlatFlow.SourceGenerators

Pipelines (high-throughput System.IO.Pipelines support, .NET 8+ only):

dotnet add package FlatFlow.Pipelines

Most projects only need FlatFlow. Add FlatFlow.SourceGenerators to opt into compile-time code generation, and FlatFlow.Pipelines for server-side high-throughput scenarios.

Quick Start

[DelimitedRecord(',')]
[HasHeader]
class Order
{
    [Field(1)] public int Id { get; set; }
    [Field(2)] public string ProductName { get; set; } = "";
    [Field(3)] public int Quantity { get; set; }
    [Field(4)] public decimal UnitPrice { get; set; }
}

using var reader = new FlatReader<Order>(File.OpenRead("orders.csv"));
foreach (var order in reader.Read())
    Console.WriteLine($"{order.Id} — {order.ProductName} x{order.Quantity}");

Packages

Defining Schemas

Attribute-based

// Delimited
[DelimitedRecord(',')]
[HasHeader]
public class Order
{
    [Field(1)] public int Id { get; set; }
    [Field(2)] public string Name { get; set; } = "";
    [Field(3), FieldConverter(typeof(IsoDateConverter))] public DateTime CreatedAt { get; set; }
    [FieldIgnored] public string Internal { get; set; } = "";
}

// Fixed-length
[FixedLengthRecord]
public class Transaction
{
    [Field(1, Width = 10, Align = RecordAlignment.Right, Padding = '0')] public int Id { get; set; }
    [Field(2, Width = 30, Align = RecordAlignment.Left)]                  public string Description { get; set; } = "";
    [Field(3, Width = 8)]                                                 public DateTime Date { get; set; }
}

Fluent builder

// Delimited
var schema = DelimitedSchema.Define(opts => opts.Delimiter(',').HasHeader())
    .AddField<int>("Id")
    .AddField<string>("Name")
    .AddField<DateTime>("CreatedAt", f => f.UseConverter(new IsoDateConverter()));

// Fixed-length
var schema = FixedLengthSchema.Define()
    .AddField<int>("Id",             f => f.Width(10).AlignRight().Pad('0'))
    .AddField<string>("Description", f => f.Width(30).AlignLeft())
    .AddField<DateTime>("Date",      f => f.Width(8));

Schemas are immutable once built — thread-safe and shareable across readers and writers.

Reading & Writing

// Attribute-derived schema
using var reader = new FlatReader<Order>(stream);

// Explicit schema
using var reader = new FlatReader<Order>(stream, schema);

// Sync
foreach (var order in reader.Read()) { }

// Async
await foreach (var order in reader.ReadAsync()) { }

// Writing
using var writer = new FlatWriter<Order>(stream);
writer.Write(orders);
await writer.WriteAsync(orders);

Source Generators

Opt in with [GenerateFlatReader] / [GenerateFlatWriter] on a partial class. At compile time FlatFlow generates concrete reader and writer classes — no reflection, trimming-safe, Native AOT compatible.

[GenerateFlatReader]
[GenerateFlatWriter]
[DelimitedRecord(',')]
public partial class Order { ... }

// Generated: OrderFlatReader : FlatReaderBase<Order>
//            OrderFlatWriter : FlatWriterBase<Order>

using var reader = new OrderFlatReader(stream);
foreach (var order in reader.Read()) { }

Schema errors (duplicate fields, width mismatches) become compiler errors rather than runtime exceptions.

Error Handling

FlatFlow gives you three ways to deal with bad records.

Throw (default) — stops on the first error:

using var reader = new FlatReader<Order>(stream);
foreach (var order in reader.Read()) { } // throws FlatFlowException on bad record

ReadResults() — a discriminated union; every record is either a value or an error, never silently dropped:

foreach (var result in reader.ReadResults())
{
    if (result.IsSuccess)
        db.Insert(result.Value);
    else
        log.Warn($"Line {result.Error!.LineNumber}: {result.Error.Message}");
}

// Async variant
await foreach (var result in reader.ReadResultsAsync()) { }

OnError callback — collect or skip bad records and continue reading:

var options = new ReaderOptions { OnError = _ => ErrorAction.Collect };
using var reader = new FlatReader<Order>(stream, options);
var orders = reader.Read().ToList();
IReadOnlyList<FlatFlowError> errors = reader.Errors;

Hierarchical Files

For files where multiple record types appear in a defined sequence — file headers, batch groups, detail lines, trailers — FileLayout describes the structure and HierarchicalFlatReader enforces and assembles it. This is the pattern used by formats like CNAB 240.

var layout = FileLayout.Define()
    .Record<FileHeader>(r => r.Optional().IdentifiedBy(line => line[0] == '0'))
    .Group(g => g
        .Optional().Many()
        .Record<BatchHeader>(r => r.Optional().IdentifiedBy(line => line[0] == '1'))
        .Record<Detail>(r => r.Required().Many().IdentifiedBy(line => line[0] == '3'))
        .Record<BatchTrailer>(r => r.Optional().IdentifiedBy(line => line[0] == '5')))
    .Record<FileTrailer>(r => r.Optional().IdentifiedBy(line => line[0] == '9'));

Aggregated reading — loads the full hierarchy into memory:

using var reader = new HierarchicalFlatReader(stream, layout);
FileResult result = reader.Read();

result.Get<FileHeader>();              // FileHeader?
result.Groups[0].GetMany<Detail>();   // IReadOnlyList<Detail>
result.Errors;                         // all collected errors

Streaming reading — yields records one at a time for large files:

foreach (var entry in reader.ReadFlattened())
{
    switch (entry)
    {
        case FlatEntry<FileHeader> h:  ...; break;
        case FlatEntry<Detail> d:      ...; break;
        case FlatEntry<FileTrailer> t: ...; break;
    }
}

Writing:

using var writer = new HierarchicalFlatWriter(stream, layout);
writer.Write(w =>
{
    w.Write(new FileHeader { ... });
    foreach (var batch in batches)
    {
        w.BeginGroup();
        w.Write(new BatchHeader { ... });
        foreach (var detail in batch.Details) w.Write(detail);
        w.Write(new BatchTrailer { ... });
        w.EndGroup();
    }
    w.Write(new FileTrailer { ... });
});

Cardinality rules:

Audit mode — reads the entire file regardless of errors, then reports everything:

using var reader = new HierarchicalFlatReader(stream, layout, LayoutOptions.AuditMode);
FileResult result = reader.Read();

foreach (var error in result.Errors)
    Console.WriteLine($"Line {error.LineNumber} [{error.Category}] {error.Message}");

Raw Access

The lowest layer returns ReadOnlySpan<char> fields directly — zero allocation in the hot path:

using var raw = new RawFlatReader(stream, schema);
foreach (var record in raw.Read())
{
    ReadOnlySpan<char> id   = record[0];
    ReadOnlySpan<char> name = record[1];
    // parse without allocating strings
}

Target Frameworks

.NET Framework is not supported.