Wolfgang.Etl.Xml 0.8.1

Wolfgang.Etl.Xml

Extractors and loaders for working with XML files using the Wolfgang.Etl design pattern

📦 Installation

dotnet add package Wolfgang.Etl.Xml

📄 License

This project is licensed under the MIT License. See the LICENSE file for details.

📚 Documentation

• GitHub Repository: https://github.com/Chris-Wolfgang/ETL-Xml
• API Documentation: https://Chris-Wolfgang.github.io/ETL-Xml/
• Formatting Guide: README-FORMATTING.md
• Contributing Guide: CONTRIBUTING.md

🚀 Quick Start

Extract items from an XML stream through a full ETL pipeline:

using Wolfgang.Etl.TestKit;
using Wolfgang.Etl.Xml;

// Extract from XML → Transform → Load into memory
var extractor = new XmlSingleStreamExtractor<Person>(xmlStream);

var transformer = new TestTransformer<Person>();
var loader = new TestLoader<Person>(collectItems: true);

await loader.LoadAsync(transformer.TransformAsync(extractor.ExtractAsync()));

var items = loader.GetCollectedItems();

Load items to an XML stream from an in-memory source:

// Extract from memory → Transform → Load to XML
var extractor = new TestExtractor<Person>(people);
var transformer = new TestTransformer<Person>();

var loader = new XmlSingleStreamLoader<Person>(outputStream);

await loader.LoadAsync(transformer.TransformAsync(extractor.ExtractAsync()));

Fluent pipeline (EtlPipeline)

For end-to-end wiring, the XML source and sink factories plug straight into the EtlPipeline chain — no explicit extractor, transformer, or loader variables. The factories read the same as the CSV/JSON siblings:

using Wolfgang.Etl.Abstractions;
using Wolfgang.Etl.Xml;

// Read a single-root XML file → write a single-root XML file.
await EtlPipeline
    .Create()
    .XmlSingleStreamExtractor<Person>("people.xml")
    .XmlSingleStreamLoader<Person>("people-copy.xml")
    .RunAsync();

Insert a Through stage to transform or filter records mid-stream:

await EtlPipeline
    .Create()
    .XmlSingleStreamExtractor<Person>("people.xml")
    .Through<Person>(people => people.Where(p => p.Age >= 18))
    .XmlSingleStreamLoader<Person>("adults.xml")
    .RunAsync(progress, cancellationToken);

Mix and match the single- and multi-stream shapes — e.g. fan a single XML document out to one file per record:

await EtlPipeline
    .Create()
    .XmlSingleStreamExtractor<Person>(sourceStream)
    .XmlMultiStreamLoader<Person>(person => File.Create($"{person.LastName}.xml"))
    .RunAsync();

…or fan the mirror direction — merge many single-document XML files back into one root document:

await EtlPipeline
    .Create()
    .XmlMultiStreamExtractor<Person>(Directory.EnumerateFiles("inbox", "*.xml").Select(File.OpenRead))
    .XmlSingleStreamLoader<Person>("people.xml")
    .RunAsync();

Stream ownership: path-based factories own the file stream they open and close it when the run finishes — on success and failure. Stream-based factories leave the caller's stream alone (honouring XmlSingleStream…Options.LeaveOpen), so the caller controls its lifetime.

Compressed streams (.xml.gz)

Every extractor and loader works against a plain Stream, so compression is transparent — wrap the underlying stream in a GZipStream (or any System.IO.Compression codec):

using System.IO.Compression;

// Write gzip-compressed XML. LeaveOpen = false lets the loader dispose the
// GZipStream when the load completes, flushing the gzip footer.
using (var file = File.Create("people.xml.gz"))
using (var gzip = new GZipStream(file, CompressionMode.Compress))
{
    var loader = new XmlSingleStreamLoader<Person>(gzip);
    await loader.LoadAsync(people);
}

// Read it back — decompress on the way in.
using (var file = File.OpenRead("people.xml.gz"))
using (var gunzip = new GZipStream(file, CompressionMode.Decompress))
{
    var extractor = new XmlSingleStreamExtractor<Person>(gunzip);
    await foreach (var person in extractor.ExtractAsync())
    {
        // ...
    }
}

See the runnable CompressedStreamRoundTripAsync example in examples/Wolfgang.Etl.Xml.Examples.

XSD validation during extraction

XmlSingleStreamExtractor<T> accepts a custom XmlReaderSettings (and clones it before use), so an XSD is validated as the document is read — no extra pass:

using System.Xml;
using System.Xml.Schema;

var schemas = new XmlSchemaSet();
schemas.Add(targetNamespace: null, "person.xsd");

var settings = new XmlReaderSettings
{
    ValidationType = ValidationType.Schema,
    Schemas = schemas,
};

var extractor = new XmlSingleStreamExtractor<Person>(stream, settings, logger);
await foreach (var person in extractor.ExtractAsync())
{
    // only schema-valid records reach here
}

A schema violation surfaces from ExtractAsync as an InvalidOperationException whose InnerException is the XmlSchemaValidationException (with the offending line and reason) — XmlSerializer wraps the reader's validation error. See the runnable XsdValidationAsync example in examples/Wolfgang.Etl.Xml.Examples.

Customizing the serialized XML

Both the extractors and loaders build an XmlSerializer from your record type (new XmlSerializer(typeof(T))) — loaders to serialize, extractors to deserialize — so they honour the standard System.Xml.Serialization attributes on your record type — element/attribute names, ignored members, the root element, collection shaping, and namespaces:

[XmlRoot("Employee")]
public sealed class Person
{
    [XmlElement("FullName")]          // <FullName> instead of <FirstName>
    public string FirstName { get; set; } = "";

    [XmlAttribute("years")]           // age as an attribute: <Employee years="30">
    public int Age { get; set; }

    [XmlIgnore]                        // never serialized
    public string InternalId { get; set; } = "";
}

Attribute-free customization (XmlAttributeOverrides) is not currently supported. Because the serializer is constructed from the type alone, there is no hook to inject an XmlAttributeOverrides instance. If you don't own the type or want to keep it attribute-free, project it onto a small DTO you do control (and decorate that), or open an issue for first-class XmlAttributeOverrides support.

Per-item error handling & dead-lettering

By default a stream/record that fails to deserialize or serialize aborts the run. On the multi-stream classes — where each stream is an independent record — you can instead skip or dead-letter the failed record and keep going by assigning an ErrorPolicy (inherited from the Abstractions base stages). Ready-made policies live in the Wolfgang.Etl.ErrorPolicies package:

using Wolfgang.Etl.ErrorPolicies;

var deadLetters = new List<ItemErrorContext>();

var extractor = new XmlMultiStreamExtractor<Person>(streams)
{
    // Skip / SkipAndLog(logger) / SkipAndDeadLetter(...) / SkipDeadLetterAndLog(...)
    ErrorPolicy = ItemErrorPolicy.SkipAndDeadLetter(deadLetters),
};

await foreach (var person in extractor.ExtractAsync())
{
    // only successfully-deserialized records reach here
}

// extractor.CurrentErrorItemCount == deadLetters.Count;
// each ItemErrorContext carries the 1-based item number and the exception.

The dead-letter policies are also overloaded for a System.Threading.Channels.ChannelWriter<ItemErrorContext>. See the runnable ErrorPolicyDeadLetterAsync example in examples/Wolfgang.Etl.Xml.Examples.

Single-stream classes keep fail-fast semantics. XmlSingleStreamExtractor<T> and XmlSingleStreamLoader<T> read/write one shared streaming document, which cannot resume mid-record after a partial failure, so a bad record aborts the run. Use the multi-stream variants for per-record error capture.

Metrics & observability

Every extractor and loader emits System.Diagnostics.Metrics measurements to the Wolfgang.Etl.Xml meter — no configuration required, and zero measurable overhead when nothing is listening. Point OpenTelemetry (or any MeterListener) at the meter to get throughput, skip/error rates, and operation latency:

using var meterProvider = Sdk.CreateMeterProviderBuilder()
    .AddMeter("Wolfgang.Etl.Xml")
    .AddPrometheusExporter()
    .Build();

Every measurement is tagged with etl.operation (extract / load), etl.component (XmlSingleStream / XmlMultiStream), and etl.record_type.

✨ Features

Extractors

• XmlSingleStreamExtractor<T> — Extracts items from a single XML stream containing a root element with child elements (e.g. <ArrayOfPerson><Person/>...</ArrayOfPerson>).
• XmlMultiStreamExtractor<T> — Extracts items from multiple XML streams, one document per stream.

Loaders

• XmlSingleStreamLoader<T> — Loads items into a single XML stream wrapped in a root element.
• XmlMultiStreamLoader<T> — Loads items into multiple XML streams via a factory function, one document per stream.
• XmlReport — Progress report returned via IProgress<XmlReport>. Properties: CurrentItemCount, CurrentSkippedItemCount.

EtlPipeline factories

Class-named factories over the fluent EtlPipeline chain, so XML sources and sinks compose without hand-wiring an extractor/loader:

• XmlSingleStreamExtractor<T>(path) / (stream, options?) — seeds a pipeline from a single-root XML source. The path overload owns and closes the file stream.
• XmlMultiStreamExtractor<T>(streams) — seeds a pipeline from a sequence of single-document XML streams (one record each).
• XmlSingleStreamLoader<T>(path, options?) / (stream, options?) — terminates a pipeline into a single-root XML document. The path overload owns and closes the file stream.
• XmlMultiStreamLoader<T>(streamFactory) — terminates a pipeline, writing one XML document per record to a per-record stream.

Constructor overloads

Each extractor and loader provides two public constructors (the first parameter varies by type):

• XmlSingleStreamExtractor<T> / XmlSingleStreamLoader<T> — (stream) or (stream, settings, logger)
• XmlMultiStreamExtractor<T> — (streams) or (streams, settings, logger) where streams is IEnumerable<Stream>
• XmlMultiStreamLoader<T> — (streamFactory) or (streamFactory, settings, logger) where streamFactory is Func<T, Stream>

Progress reporting

var extractor = new XmlSingleStreamExtractor<Person>(xmlStream);
extractor.ReportingInterval = 100; // Report every 100ms

var progress = new Progress<XmlReport>(report =>
    Console.WriteLine($"Progress: {report.CurrentItemCount} items, {report.CurrentSkippedItemCount} skipped")
);

var transformer = new TestTransformer<Person>();
var loader = new TestLoader<Person>(collectItems: true);

await loader.LoadAsync(transformer.TransformAsync(extractor.ExtractAsync(progress)));

Skip and maximum item count

var extractor = new XmlSingleStreamExtractor<Person>(xmlStream);
extractor.SkipItemCount = 10;     // Skip first 10 items
extractor.MaximumItemCount = 5;   // Then take 5 items

var transformer = new TestTransformer<Person>();
var loader = new TestLoader<Person>(collectItems: true);

await loader.LoadAsync(transformer.TransformAsync(extractor.ExtractAsync()));
// extractor.CurrentItemCount == 5, extractor.CurrentSkippedItemCount == 10

🎯 Supported Frameworks

This library targets:

• .NET Framework: 4.6.2, 4.8.1
• .NET Standard: 2.0
• .NET: 8.0, 10.0

See the NuGet package page for the authoritative per-TFM compatibility matrix.

🔍 Code Quality & Static Analysis

This project enforces strict code quality standards through 7 specialized analyzers and custom async-first rules:

Analyzers in Use

1. Microsoft.CodeAnalysis.NetAnalyzers - Built-in .NET analyzers for correctness and performance
2. Roslynator.Analyzers - Advanced refactoring and code quality rules
3. AsyncFixer - Async/await best practices and anti-pattern detection
4. Microsoft.VisualStudio.Threading.Analyzers - Thread safety and async patterns
5. Microsoft.CodeAnalysis.BannedApiAnalyzers - Prevents usage of banned synchronous APIs
6. Meziantou.Analyzer - Comprehensive code quality rules
7. SonarAnalyzer.CSharp - Industry-standard code analysis

Async-First Enforcement

This library uses BannedSymbols.txt to prohibit synchronous APIs and enforce async-first patterns:

Blocked APIs Include:

• Task.Wait(), Task.Result - Use await instead
• Thread.Sleep() - Use await Task.Delay() instead
• Synchronous file I/O (File.ReadAllText) - Use async versions
• Synchronous stream operations - Use ReadAsync(), WriteAsync()
• Parallel.For/ForEach - Use Task.WhenAll() or Parallel.ForEachAsync()
• Obsolete APIs (WebClient, BinaryFormatter)

Building from Source

Prerequisites

• .NET 10.0 SDK (older SDKs work for restoring the older TFMs but the build/test matrix targets the full range up to .NET 10.0)
• Optional: PowerShell Core for formatting scripts

Build Steps

# Clone the repository
git clone https://github.com/Chris-Wolfgang/ETL-Xml.git
cd ETL-Xml

# Restore dependencies
dotnet restore

# Build the solution
dotnet build --configuration Release

# Run tests
dotnet test --configuration Release

# Run code formatting (PowerShell Core)
pwsh ./format.ps1

Code Formatting

This project uses .editorconfig and dotnet format:

# Format code
dotnet format

# Verify formatting
dotnet format --verify-no-changes

See README-FORMATTING.md for detailed formatting guidelines.

Building Documentation

This project uses DocFX to generate API documentation:

# Install DocFX (one-time setup)
dotnet tool install -g docfx

# Generate API metadata and build documentation
cd docfx_project
docfx metadata  # Extract API metadata from source code
docfx build     # Build HTML documentation

# Documentation is generated in the docs/ folder at the repository root

The documentation is automatically built and deployed to GitHub Pages when changes are pushed to the main branch.

Local Preview:

# Serve documentation locally (with live reload)
cd docfx_project
docfx build --serve

# Open http://localhost:8080 in your browser

Documentation Structure:

• docfx_project/ - DocFX configuration and source files
• docs/ - Generated HTML documentation (published to GitHub Pages)
• docfx_project/index.md - Main landing page content
• docfx_project/docs/ - Additional documentation articles
• docfx_project/api/ - Auto-generated API reference YAML files

🤝 Contributing

Contributions are welcome! Please see CONTRIBUTING.md for:

• Code quality standards
• Build and test instructions
• Pull request guidelines
• Analyzer configuration details

Acknowledgments

• Wolfgang.Etl.Abstractions — the base class framework this library builds on
• Wolfgang.Etl.TestKit — test doubles and contract test base classes for pipeline development