SourceDocParser 0.3.1-alpha

SourceDocParserLib

Roslyn-based .NET assembly walker that turns compiled .dll + .pdb + .xml triples into a strongly-typed API catalog (types, members, signatures, XML docs, inheritdoc, SourceLink) and hands it to a pluggable emitter for rendering.

The catalog is format-neutral. Emitters decide how to render it — Markdown for Zensical / mkdocs Material, or YAML for docfx ManagedReference, with room for other targets.

Packages

Logging flows through Microsoft.Extensions.Logging.Abstractions source-generated [LoggerMessage] partials, so any host (Serilog, Console, NLog, …) plugs in without the libraries taking a dependency on a specific backend.

Quick start

var loggerFactory = LoggerFactory.Create(b => b.AddConsole());

var source = new NuGetAssemblySource(
    rootDirectory: "/path/to/repo",   // contains nuget-packages.json
    apiPath:       "/path/to/api",    // where lib/ + refs/ get extracted
    logger:        loggerFactory.CreateLogger<NuGetAssemblySource>());

var emitter = new ZensicalDocumentationEmitter();

var result = await new MetadataExtractor().RunAsync(
    source,
    outputRoot: "/path/to/markdown-output",
    emitter,
    loggerFactory.CreateLogger<MetadataExtractor>());

Console.WriteLine($"Emitted {result.PagesEmitted} pages across {result.CanonicalTypes} types.");

Performance

The pipeline is built around a span-based XML scanner, pooled buffers, eager release of memory-mapped reference DLLs, and a streaming type merger that consumes catalogs as they land. The result is a small, predictable allocation budget and a fast wall-time per assembly.

Benchmark workload. Numbers below are from the BenchmarkDotNet suite under src/benchmarks/, run on a Ryzen 7 5800X / .NET 10. The workload extracts three real NuGet packages from nuget.org — pulling each package's lib/ and ref/ trees and the matching reference assemblies, walking every public symbol across ~19 target-framework groups, parsing the shipped XML doc files for each assembly, resolving <inheritdoc/> chains, and emitting roughly 600 canonical type pages after cross-TFM merge. The local NuGet cache is warmed once during global setup so per-iteration timings measure the walk + merge + emit pipeline, not the network leg.

End-to-end (MetadataExtractor.RunAsync):

Peak working set is bounded too: per-TFM compilation loaders dispose as soon as their last assembly finishes walking, so the memory-mapped BCL reference views are released eagerly instead of accumulating until RunAsync exits.

Per-call hotspots:

Emitter cost per type page (no I/O, just markup formatting; baseline = Zensical Markdown):

DocFx YAML is heavier by design — every member duplicates uid / commentId / parent / name / nameWithType / fullName, and the page-level references: list adds another mapping per cross-referenced type. The emitter still hand-writes its YAML directly via StringBuilder (no YamlDotNet runtime dependency), with a single-allocation fast path for the qualified-name composites (type.Name + "." + member.Name) that round-trips identifiers as plain scalars when escape-safe.

Side-by-side against dotnet docfx metadata. Two fully isolated standalone benchmark assemblies — benchmarks/Docfx.StandaloneBenchmarks/ (calls DotnetApiCatalog.GenerateManagedReferenceYamlFiles in-process) and benchmarks/SourceDocParser.Docfx.StandaloneBenchmarks/ (drives our pipeline through DocfxYamlEmitter) — both target the same 4 NuGet packages (ReactiveUI, Splat, DynamicData, System.Reactive), measured by BenchmarkDotNet's [ShortRunJob] on the same machine:

The two pipelines aren't strictly walking identical inputs — docfx loads a synthesised Fixture.csproj that pulls the 4 packages as transitive PackageReferences and walks one effective TFM, while our pipeline resolves every shipped lib//ref/ slice across ~19 supported TFMs from nuget-packages.json and merges across them. Working backward from that fixture difference, our per-TFM walk explains both the wall-time delta and the allocation gap (each TFM spins a fresh Roslyn compilation graph, and the cross-TFM merger holds catalogs while it dedupes UIDs). The contract pinned by the comparison is parity output (every T:, M:, P:, E: UID docfx emits, our pipeline emits too) at the per-page emit cost shown in the per-page table above.

Strategies the pipeline uses

• Custom span-based XML scanner. Every NuGet package ships an <assembly-name>.xml doc file alongside its .dll, holding the /// doc comments for every public symbol. The walker has to read each member's XML fragment per symbol, render its <see> / <c> / <list> / <inheritdoc> tags into Markdown, and do the same again per <param> / <exception> inside it — for thousands of symbols per assembly. XmlReader works for that, but its XmlTextReaderImpl allocates multi-KB internal buffers (NodeData[], NamespaceManager, char buffers, Entry[]) per construction, which dominates the doc-parse profile. So the pipeline ships a small ref struct DocXmlScanner that walks the doc text directly over ReadOnlySpan<char> and implements just the XML grammar that /// doc comments actually use. Both the per-symbol parser and the Markdown renderer drive the scanner, so per-element XML processing is allocation-free apart from the result string.
• Build-once-then-read-many XmlDocSource. Each .xml doc file is read once via File.ReadAllBytes + Encoding.UTF8.GetString, then indexed by per-member (offset, length) ranges. The substring is only materialised when a consumer calls Get(memberId), and the source is safe for concurrent reads under the parallel walker.
• Eager per-group loader disposal. Each TFM group has its own CompilationLoader with a private MetadataReferenceCache holding memory-mapped views of every reference DLL. As soon as the last assembly in a group finishes its walk, an interlocked counter drops to zero and the loader disposes — peak working set scales with the slowest-finishing group, not the total number of groups times their references.
• Streaming type merger. The parallel walk feeds ApiCatalogs into StreamingTypeMerger one at a time and immediately drops its reference, instead of accumulating every catalog in a ConcurrentBag until the walk phase finishes.
• Capture-free parallel dispatch. The Parallel.ForEachAsync lambda is static — every dependency it touches is bundled into a WalkContext record attached to each work item, so dispatch never allocates a closure object per assembly.
• Pooled StringBuilder on the converter. XmlDocToMarkdown is per-walk by construction; reusing a single builder across every Convert call eliminates the per-element allocation that would otherwise dominate the renderer.
• Pre-sized buffers. Each nupkg zip entry is sized to its known uncompressed length up front so the backing byte[] is allocated once at the right size instead of doubling-and-copying on every Write. SourceLink URL rewriting fuses the base URL and the line anchor into one interpolated-string handler call so the GitHub / Bitbucket / GitLab / Azure DevOps blob URL is materialised in a single string.

Repository layout

SourceDocParserLib/
  src/
    SourceDocParser/
    SourceDocParser.NuGet/
    SourceDocParser.Docfx/
    SourceDocParser.Zensical/
    tests/
      SourceDocParser.Tests/             unit tests (TUnit)
      SourceDocParser.IntegrationTests/  end-to-end + Zensical render-smoke
    Directory.Build.props                shared lib config
    Directory.Packages.props             central package versions
    SourceDocParserLib.slnx
  Directory.Build.props
  version.json                           Nerdbank.GitVersioning
  .editorconfig
  stylecop.json

dotnet build from src/ packs every non-test project into artifacts/packages/ automatically (<GeneratePackageOnBuild>true</GeneratePackageOnBuild>). Consumers in other repos can wire that directory up as a local feed via nuget.config until the libraries are published.

Acknowledgements

The metadata extraction pipeline is inspired by — and lifts patterns from — dotnet/docfx (MIT licensed). docfx's Roslyn-based assembly walker, inheritdoc resolution, and overall metadata model shaped this library's design. See LICENSE for the original docfx attribution.

Built on:

• Roslyn (Microsoft.CodeAnalysis.CSharp) for compilation + symbol model
• ICSharpCode.Decompiler for transitive reference resolution
• NuGet.Frameworks + NuGet.Versioning for proper TFM compatibility and SemVer ordering
• Polly v8 for HTTP retry/rate-limit pipelines

License

MIT — see LICENSE for the full text and the docfx attribution.