XmlAuditfile 0.0.7

auditfile-dotnet

Streaming reader for Dutch tax audit files (XAF and related), following auditfile-spec.

Status: draft (0.x). Not affiliated with the Belastingdienst or any software vendor.

Packages

Write your own ISink to load into a database or anything else; only the reader package is needed then.

using XmlAuditfile;
using XmlAuditfile.Memory;

var sink = new MemorySink();
var source = AuditfileSource.FromFactory(() => File.OpenRead("administratie.xaf"), "administratie.xaf");
var report = await Auditfile.ReadAsync(source, sink);

foreach (var diagnostic in report.Diagnostics)
{
    Console.WriteLine(diagnostic);
}

foreach (var line in sink.Rows("tr_line"))
{
    Console.WriteLine($"{line.GetString("acc_id")} {line.GetDecimal("amnt")} {line.GetString("amnt_tp")}");
}

Rows have typed accessors (GetString, GetDecimal, GetInteger, GetDate, GetDateTime, GetTime, GetBoolean) that check the column type and name the problem; TryGetValue tells whether a column exists in this version. GetDecimal also reads an integer column, since a column can be integer in one version and decimal in another. In a loop over many rows, look the column up once (table.FindColumn("amnt")) and pass the SchemaColumn: no lookup by name per row.

Values are read in XSD notation only: a date-time such as 03/01/2017 10:00:00 is refused (XAF-S004) rather than guessed.

What it does

• Determines the version: forced in code, official namespace or alias from the registry, optionally derived from the namespace (UseResolver). Anything else stops with a clear diagnostic.
• Reads in two phases: scan (validate against the built-in XSD, count rows per table) and load (stream rows in batches to a sink). Verified mode loads only a valid file.
• Turns every repeating element into a table following the manifest generated by auditfile-spec. Rows know their id, parent, position and source part; values are typed (decimal, long, DateTime, ...).
• Checks required and expected tables, keys, references and the XAF business rules (balanced transactions, header counts and totals).
• Reads split audit files (audit.xaf, audit-2.xaf, ...) as one file; you pass the parts in order.
• Reports everything as diagnostics with stable codes, in Dutch or English.

Entry points

Every call takes a CancellationToken, checked in the scan phase as well as the load phase. ReadOptions refuses values out of range (BatchSize, MaxDiagnostics, MaxDepth, MaxValueLength at least 1) with an ArgumentOutOfRangeException. When a sink fails and its AbortAsync fails too, the read throws an AggregateException with both errors. | AuditfileVersion.Financial.V3_2 | Manifest, XSD, definitions and dialects of a version | | Auditfile.Dialects.Add(version, json) / Auditfile.Views.Add(json) | Add your own dialects and views next to the built-in ones (see below) | | AuditfileSchema.ToJson() / FromJson() | Store and restore the schema of an import |

Sources: the reader never opens files; you supply the streams.

// Files on disk, e.g. audit.xaf and audit-2.xaf
var source = AuditfileSource.FromFactories(paths.Select(p => (Path.GetFileName(p), (Func<Stream>)(() => File.OpenRead(p)))).ToArray());

Custom dialects and views

Dialects and views come from auditfile-spec and are embedded in the package. You can add your own at runtime, in the same format as dialects.json and views/<type>.json of the spec, without a fork or a custom build. Pass the JSON text or a UTF-8 stream; the reader opens no files, so open them yourself.

// { "formatVersion": 1, "dialects": [ { "name": "my-package", "match": { ... }, "corrections": [ ... ] } ] }
Auditfile.Dialects.Add(AuditfileVersion.Financial.V3_2, File.ReadAllText("dialects.json"));

// { "formatVersion": 1, "type": "xaf", "views": [ { "name": "lines_per_account", ... } ] }
using (var views = File.OpenRead("views.json"))
{
    Auditfile.Views.Add(views);
}

var report = await Auditfile.ReadAsync(source, sink);      // recognizes and applies the added dialect
var result = sink.Run(sink.Schema!.Version!.FindView("lines_per_account")!);

• Added definitions come after the built-in ones in AuditfileVersion.Dialects and Views, so FindView, version resolution and dialect corrections use them like the built-in ones. A view is added for every version of its type; whether it fits a version is decided when it runs (XAF-Q001), as for built-in views. ApplyDialects = false ignores added dialects too.
• A file is checked when it is added and refused as a whole when anything is wrong (FormatException): invalid JSON, a formatVersion other than 1, an unknown dialect action, view aggregate or operator, or a correction code that is not a dialect diagnostic (XAF-C001–C003). An unknown audit file type or a name the version or type already has throws an ArgumentException. Built-in definitions cannot be replaced or removed.
• Auditfile.Dialects.Remove(version, name) and Auditfile.Views.Remove(type, name) remove one added definition; Clear() removes all of them. The registries are thread-safe and shared by every read in the process; a read that has started keeps the dialects it found.

Targets

netstandard2.0 (.NET Framework 4.7.2+) and net10.0. No package dependencies on any target. On net10.0 the library is trimming and Native AOT compatible; the trim and AOT analyzers run in every build. Example programs live in auditfile-dotnet-samples.

WebAssembly: the library uses no file system, threads or reflection; all input comes from your streams. Browser file streams are asynchronous only, so copy the file into memory first and use AuditfileSource.FromFactory(() => new MemoryStream(bytes)).

Web applications: the XML is parsed synchronously; only the sink is called asynchronously. Parsing is CPU-bound, and asynchronous reads per node would only slow it down. ASP.NET Core does not allow synchronous reads of the request body, so do not pass Request.Body to FromStream: copy the upload to a temporary file and read that. A large file keeps a thread busy while it is read, so run big imports in a background job rather than in the request.

var path = Path.GetTempFileName();
try
{
    await using (var file = File.Create(path))
    {
        await upload.CopyToAsync(file, cancellationToken); // upload: IFormFile
    }

    var report = await Auditfile.ValidateAsync(
        AuditfileSource.FromFactory(() => File.OpenRead(path), upload.FileName), cancellationToken: cancellationToken);
}
finally
{
    File.Delete(path);
}

Diagnostics

Every finding is a diagnostic with a stable code from diagnostics.json of auditfile-spec. The report lists at most MaxDiagnostics (default 1000), followed by XAF-X900 when more were found. A full list makes room for errors by leaving out infos and warnings, and fatal diagnostics are always included. IsValid, HasErrors, HasFatal and Count(severity) count every diagnostic, also the ones left out; IsTruncated says whether some were.

Defined in the spec but not checked yet: S007 and S009 (in verified mode the XSD reports too many occurrences and values outside a code list as D001), R005–R016 (ascending numbers, periods, eleven test, IBAN, personal data), X005 (the XML parser reports an invalid character as X001) and E001–E004 (envelopes).

Layout

Definitions

Everything in generated/ of a pinned auditfile-spec release (XSD, manifests, definitions, dialects, registry, diagnostics) is embedded in the package. The version objects (AuditfileVersion) are part of this reader; a test checks that they match the registry.

The spec is not committed. spec.lock.json pins a release (tag and commit) of auditfile-spec, and tools/fetch-spec.ps1 copies only generated/ of that commit into spec/generated/ (git-ignored; a plain folder, not a git checkout). The build stops with a clear error when spec/ is missing.

pwsh tools/fetch-spec.ps1        # after cloning and whenever spec.lock.json changes
dotnet build XmlAuditfile.slnx
dotnet test --solution XmlAuditfile.slnx

To build against a local spec checkout instead: dotnet build -p:SpecRoot=../auditfile-spec/.

Workflows

License

Apache License 2.0.