Syfuhs.Wikipedia 1.0.3

Syfuhs.Wikipedia

A streaming, strongly typed .NET parser for Wikipedia and Wikidata dumps. Designed for multi-gigabyte files: forward-only parsing in near-constant memory, filter pushdown that skips page text before it's decoded, multistream support for random access and parallel scans, tolerant wikitext parsing, and map-ready geographic extraction.

Reading Wikipedia dumps

Stream every page:

await using var archive = WikiArchive.Open(@"D:\dumps\enwiki-latest-pages-articles.xml.bz2");

Console.WriteLine(archive.SiteInfo?.SiteName);

await foreach (var page in archive.Pages())
{
    Console.WriteLine($"{page.Id}: {page.Title}");
}

Filter inside the parser (rejected pages are skipped before their text is decoded):

var filter = new PageFilter
{
    Namespaces = [WikiNamespace.Main],
    IncludeRedirects = false,
    TitlePredicate = static t => t.Contains("County"),
};

await foreach (var page in archive.Pages(filter))
{
    // main-namespace, non-redirect pages with "County" in the title
}

Random access and parallel scans over multistream dumps:

await using var archive = WikiArchive.OpenMultistream(
    @"D:\dumps\enwiki-latest-pages-articles-multistream.xml.bz2",
    @"D:\dumps\enwiki-latest-pages-articles-multistream-index.txt.bz2");

WikiPage? page = await archive.GetPageAsync("Adams County, Washington");

await foreach (var p in archive.Pages(filter, new ParallelOptions { MaxDegreeOfParallelism = 8 }))
{
    // unordered, one decompression worker per core
}

Partitioned dumps (the newer Wikimedia layout: many ...-pXpY.xml.bz2 files, no index):

var set = WikiArchiveSet.Open(@"D:\dumps\enwiki-2026-08-01");

WikiPage? page = await set.GetPageAsync(91253);      // routed by pXpY file-name range

await foreach (var p in set.Pages(filter, new ParallelOptions { MaxDegreeOfParallelism = 8 }))
{
    // one worker per partition file
}

Plain .xml, .xml.gz, and .xml.bz2 inputs are all detected from content, not extension.

Parsing wikitext

Parse a page's wikitext into queryable structure (the Syfuhs.Wikipedia.Wikitext namespace):

using Syfuhs.Wikipedia.Wikitext;

WikitextDocument doc = page.ParseWikitext();

var infobox = doc.Infoboxes.FirstOrDefault();       // {{Infobox settlement}} fields
var coord   = doc.PrimaryCoordinate;                 // {{Coord}} -> decimal degrees
var cats    = doc.Categories;                        // [[Category:...]] with sort keys
var history = doc.Section("History")?.GetPlainText();
var summary = doc.GetLeadSummary();                  // first lead paragraph, plaintext

if (doc.IsRedirect) Console.WriteLine(doc.RedirectTarget);

Sections (with hierarchy), templates (recursive, positional + named parameters), links, images with captions (galleries included), tables (rows/cells/headers with key-value row access), lists, <ref> citations, and markup-stripped plaintext are all exposed; parsing is tolerant and never throws on malformed markup.

foreach (var table in doc.Tables)
    foreach (var row in table.ToKeyValues())          // header -> plaintext cell
        Console.WriteLine($"{row["County"]}: {row["Population"]}");

Reading Wikidata dumps

Scan Wikidata entity dumps for structured data (the Syfuhs.Wikipedia.Wikidata namespace):

using Syfuhs.Wikipedia.Wikidata;

var wikidata = WikidataArchive.Open(@"D:\data\wikidata-latest-all.json.gz");
var filter = new EntityFilter { Properties = ["P625"], SitelinkSite = "enwiki" };

foreach (var entity in wikidata.Entities(filter))   // byte-level prefilter skips ~90% of lines
{
    var coord = entity.Coordinate;                   // best-rank P625, decimal degrees
    var kinds = entity.InstanceOfIds;                // P31: ["Q515"] = city
    var title = entity.GetSitelinkTitle("enwiki");   // join key back to the XML archive
}

// Resolve entity ids to display names in one pass:
var names = wikidata.GetLabels(["Q515", "Q532"]);    // {Q515: city, Q532: village}

Scan straight off an HTTP response (or any stream) without staging the dump on disk. A non-seekable stream is scanned once; a seekable one is rewound per scan:

using var response = await http.GetAsync(dumpUrl, HttpCompletionOption.ResponseHeadersRead);
using var wikidata = WikidataArchive.Open(await response.Content.ReadAsStreamAsync());

foreach (var entity in wikidata.Entities(filter))   // one forward pass, near-constant memory
{
    // ...
}

Extracting geographic data

Turn both sources into flat, indexable records (the Syfuhs.Wikipedia.Extraction namespace):

using Syfuhs.Wikipedia.Extraction;

var extractor = new GeoTagExtractor();
foreach (GeoTag tag in extractor.Extract(wikidata))
{
    // tag: coordinates, P31 kinds, P131 admin parents, P17 country,
    // population, enwiki article title - flat, ready for a spatial index.
}

// Join external references (OSM wikipedia=en:... tags) back to articles:
var (lang, title) = WikiTitle.ParseTag("en:Adams_County,_Washington")!.Value;
WikiPage? page = await archive.GetPageFollowingRedirectsAsync(title);

Long scans accept an IProgress<ScanProgress> (compressed-bytes based, so the fraction is accurate) and honor CancellationToken throughout.