Wolfgang.Etl.Json 0.8.1

Wolfgang.Etl.Json

Extractors and Loaders for reading and writing JSON, JSONL, and multi-stream JSON files, built on Wolfgang.Etl.Abstractions.

📦 Installation

dotnet add package Wolfgang.Etl.Json

NuGet Package: Wolfgang.Etl.Json

✨ Features

All components support:

• System.Text.Json serialization with optional JsonSerializerOptions
• ILogger<T> for structured diagnostic logging at Debug, Information, Warning, and Error levels
• SkipItemCount and MaximumItemCount for pagination
• Progress reporting via IProgress<TProgress> with configurable ReportingInterval
• Cancellation via CancellationToken

🚀 Quick Start

Extract from a JSON array

using var stream = File.OpenRead("people.json");
var extractor = new JsonSingleStreamExtractor<Person>(stream, logger);

await foreach (var person in extractor.ExtractAsync(cancellationToken))
{
    Console.WriteLine(person.Name);
}

Load to a JSON array

using var stream = File.Create("output.json");
var loader = new JsonSingleStreamLoader<Person>(stream, logger);

await loader.LoadAsync(items, cancellationToken);

Extract from multiple files (one object per file)

var streams = Directory.GetFiles("data/", "*.json").Select(File.OpenRead);
var extractor = new JsonMultiStreamExtractor<Person>(streams, logger);

await foreach (var person in extractor.ExtractAsync(cancellationToken))
{
    Console.WriteLine(person.Name);
}

Load to multiple files (one object per file)

var loader = new JsonMultiStreamLoader<Person>
(
    person => File.Create($"output/{person.Id}.json"),
    logger
);

await loader.LoadAsync(items, cancellationToken);

Extract from JSONL/NDJSON

using var stream = File.OpenRead("data.jsonl");
var extractor = new JsonLineExtractor<Person>(stream, logger);

await foreach (var person in extractor.ExtractAsync(cancellationToken))
{
    Console.WriteLine(person.Name);
}

Load to JSONL/NDJSON

using var stream = File.Create("output.jsonl");
var loader = new JsonLineLoader<Person>(stream, logger);

await loader.LoadAsync(items, cancellationToken);

Custom serialization options

All extractors and loaders accept an optional JsonSerializerOptions:

var options = new JsonSerializerOptions
{
    PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
    PropertyNameCaseInsensitive = true,
};

var extractor = new JsonSingleStreamExtractor<Person>(stream, options, logger);

Source generation (AOT-friendly)

All extractors and loaders accept a JsonTypeInfo<T> for reflection-free serialization:

[JsonSerializable(typeof(Person))]
internal partial class AppJsonContext : JsonSerializerContext { }

// Use the source-generated type info instead of JsonSerializerOptions
var extractor = new JsonSingleStreamExtractor<Person>(stream, AppJsonContext.Default.Person, logger);
var loader = new JsonLineLoader<Person>(stream, AppJsonContext.Default.Person, logger);

🎯 Supported Frameworks

This library targets:

• .NET Framework: 4.6.2, 4.8.1
• .NET Standard: 2.0
• .NET: 8.0, 10.0

See the NuGet package page for the authoritative per-TFM compatibility matrix.

🔍 Code Quality & Static Analysis

This project enforces strict code quality standards through 8 specialized analyzers and custom async-first rules:

Analyzers in Use

1. Microsoft.CodeAnalysis.NetAnalyzers - Built-in .NET analyzers for correctness and performance
2. Roslynator.Analyzers - Advanced refactoring and code quality rules
3. AsyncFixer - Async/await best practices and anti-pattern detection
4. Microsoft.VisualStudio.Threading.Analyzers - Thread safety and async patterns
5. Microsoft.CodeAnalysis.BannedApiAnalyzers - Prevents usage of banned synchronous APIs
6. Meziantou.Analyzer - Comprehensive code quality rules
7. SonarAnalyzer.CSharp - Industry-standard code analysis
8. Microsoft.CodeAnalysis.PublicApiAnalyzers - Tracks the public API surface to catch unintended breaking changes

Async-First Enforcement

This library uses BannedSymbols.txt to prohibit synchronous APIs:

• Task.Wait(), Task.Result - Use await instead
• Thread.Sleep() - Use await Task.Delay() instead
• Synchronous file I/O (File.ReadAllText) - Use async versions
• Synchronous stream operations - Use ReadAsync(), WriteAsync()
• Obsolete APIs (WebClient, BinaryFormatter)

🛠️ Building from Source

Prerequisites

• .NET 10.0 SDK or later
• Optional: PowerShell Core for formatting scripts

Build Steps

# Clone the repository
git clone https://github.com/Chris-Wolfgang/ETL-Json.git
cd ETL-Json

# Restore dependencies
dotnet restore

# Build the solution
dotnet build --configuration Release

# Run tests
dotnet test --configuration Release

Code Formatting

This project uses .editorconfig and dotnet format:

dotnet format
dotnet format --verify-no-changes

Building Documentation

This project uses DocFX to generate API documentation:

# Install DocFX (one-time setup)
dotnet tool install -g docfx

# Generate API metadata and build documentation
cd docfx_project
docfx metadata  # Extract API metadata from source code
docfx build     # Build HTML documentation

# Documentation is generated in the docs/ folder at the repository root

The documentation is automatically built and deployed to GitHub Pages when changes are pushed to the main branch.

Local Preview:

# Serve documentation locally (with live reload)
cd docfx_project
docfx build --serve

# Open http://localhost:8080 in your browser

🔧 Advanced Usage

Compressed streams

All extractors and loaders accept any Stream, including compression wrappers — no special API is needed. Wrap the underlying stream in a GZipStream, BrotliStream, or DeflateStream and pass it to the constructor.

See docs/COMPRESSED-STREAMS.md for examples and a TFM compatibility table.

Schema customization

Customize JSON serialization without modifying — or even owning — the POCO class: use naming policies, DefaultJsonTypeInfoResolver modifiers (.NET 8+), custom converters, or source-generated JsonTypeInfo<T> for AOT-safe serialization.

See docs/SCHEMA-CUSTOMIZATION.md for recipes.

🔍 Verify the Build

The NuGet packages published from this repository are reproducible and supply-chain-verified. See docs/REPRODUCIBLE-BUILD.md for step-by-step instructions to independently reproduce and verify any release artifact.

🤝 Contributing

Contributions are welcome! Please see CONTRIBUTING.md for:

• Code quality standards
• Build and test instructions
• Pull request guidelines
• Analyzer configuration details

📄 License

This project is licensed under the MIT License. See the LICENSE file for details.

📚 Documentation

• GitHub Repository: https://github.com/Chris-Wolfgang/ETL-Json
• API Documentation: https://Chris-Wolfgang.github.io/ETL-Json/
• Contributing Guide: CONTRIBUTING.md