Wolfgang.Etl.Transformers 0.5.2

Wolfgang.Etl.Transformers

A collection of generic, composable transformers for ETL pipelines built on Wolfgang.Etl.Abstractions.

Installation

dotnet add package Wolfgang.Etl.Transformers

Quick Start

using Wolfgang.Etl.Transformers;

// Build a pipeline: parse → filter → project → buffer
var pipeline = new SelectTransformer<string, Order>(ParseOrder)
    .Then(new WhereTransformer<Order>(o => o.IsValid))
    .Then(new SelectTransformer<Order, InvoiceRow>(ToInvoice));

await foreach (var row in pipeline.TransformAsync(rawLines))
{
    await loader.LoadAsync(row);
}

// Or buffer inline between pipeline stages:
var buffered = rawLines.Buffered(capacity: 500);
var filtered = new WhereTransformer<Order>(o => o.IsValid);
var projected = new SelectTransformer<Order, InvoiceRow>(ToInvoice);

await foreach (var row in projected.TransformAsync(filtered.TransformAsync(buffered)))
{
    await loader.LoadAsync(row);
}

Transformers

LINQ-style

Pipeline infrastructure

Composition

Pipeline operators

Referencing this package lights up LINQ-flavored operators on IEtlPipeline<T> (the pipeline core from Wolfgang.Etl.Abstractions). Each operator is a thin wrapper over the matching transformer, so they slot between the source (From(...)) and sink (To(...)) stages of a fluent pipeline:

using Wolfgang.Etl.Transformers;

await EtlPipeline
    .Create()
    .From(records)
    .Where(r => r.Amount > 0)
    .Select(r => r.Id)
    .Distinct()
    .Chunk(500)
    .To(loader)
    .RunAsync();

Data-shape operators: Where, Select, SelectMany (each with sync and async overloads), Distinct, DistinctBy, Take, Skip, TakeWhile, SkipWhile, Chunk, Buffered, Cast, and OfType.

Observability operators (watch or pace the stream without changing its shape): Tap (sync/async side effect per item, passed through unchanged); Log (Log(format, sink) — one formatted message per item via a delegate sink, no logging-framework dependency); Throttle (Throttle(minInterval) — paces items at least a TimeSpan apart to rate-limit a downstream sink).

🎯 Supported Frameworks

This library targets:

• .NET Framework: 4.6.2, 4.7.2, 4.8, 4.8.1
• .NET Standard: 2.0
• .NET: 5.0, 6.0, 7.0, 8.0, 9.0, 10.0

See the NuGet package page for the authoritative per-TFM compatibility matrix.

Code Quality & Static Analysis

This project enforces strict code quality standards through 7 specialized analyzers and custom async-first rules:

Analyzers in Use

1. Microsoft.CodeAnalysis.NetAnalyzers — Correctness, performance, and security rules
2. Roslynator.Analyzers — 500+ refactoring and code quality rules
3. AsyncFixer — Async/await best practices and anti-pattern detection
4. Microsoft.VisualStudio.Threading.Analyzers — Thread safety and async patterns
5. Microsoft.CodeAnalysis.BannedApiAnalyzers — Blocks synchronous APIs listed in BannedSymbols.txt
6. Meziantou.Analyzer — Comprehensive code quality checks
7. SonarAnalyzer.CSharp — Industry-standard code analysis

Async-First Enforcement

This library prohibits synchronous blocking calls via BannedSymbols.txt:

// ❌ Banned
task.Wait();
task.Result;
File.ReadAllText(path);
Thread.Sleep(1000);

// ✅ Required
await task;
await File.ReadAllTextAsync(path);
await Task.Delay(1000);

Building from Source

Prerequisites

• .NET 10.0 SDK or later (required to build the full TFM matrix)
• Optional: PowerShell Core for formatting scripts

Build Steps

# Clone the repository
git clone https://github.com/Chris-Wolfgang/ETL-Transformers.git
cd ETL-Transformers

# Restore dependencies
dotnet restore

# Build (Release enforces all analyzers as errors)
dotnet build --configuration Release

# Run tests
dotnet test --configuration Release

# Format code
pwsh ./scripts/format.ps1

Building Documentation

This project uses DocFX for API documentation:

dotnet tool install -g docfx
cd docfx_project
docfx build --serve
# Open http://localhost:8080

Documentation is automatically built and deployed to GitHub Pages when a GitHub Release is published.

Documentation: https://Chris-Wolfgang.github.io/ETL-Transformers/

Verify the build

This package is built reproducibly: rebuilding a tagged release with the pinned SDK produces byte-for-byte identical assemblies and packages. Every release attaches a reproducible-build-manifest.json with the expected SHA-256 of each artifact, and CI proves cross-OS reproducibility on every pull request.

To confirm a published release was built from source — and to publish your own independent verification attestation — follow the step-by-step guide in REPRODUCIBLE-BUILD.md.

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for code quality standards, build instructions, and pull request guidelines.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Acknowledgments

Built on Wolfgang.Etl.Abstractions — the base class library providing ExtractorBase<TSource, TProgress>, LoaderBase<TDestination, TProgress>, and TransformerBase<TSource, TDestination, TProgress>.