z3l.Scraper 0.1.0-preview.6

This is a prerelease version of z3l.Scraper.
dotnet add package z3l.Scraper --version 0.1.0-preview.6
                    
NuGet\Install-Package z3l.Scraper -Version 0.1.0-preview.6
                    
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="z3l.Scraper" Version="0.1.0-preview.6" />
                    
For projects that support PackageReference, copy this XML node into the project file to reference the package.
<PackageVersion Include="z3l.Scraper" Version="0.1.0-preview.6" />
                    
Directory.Packages.props
<PackageReference Include="z3l.Scraper" />
                    
Project file
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package.
paket add z3l.Scraper --version 0.1.0-preview.6
                    
#r "nuget: z3l.Scraper, 0.1.0-preview.6"
                    
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
#:package z3l.Scraper@0.1.0-preview.6
                    
#:package directive can be used in C# file-based apps starting in .NET 10 preview 4. Copy this into a .cs file before any lines of code to reference the package.
#addin nuget:?package=z3l.Scraper&version=0.1.0-preview.6&prerelease
                    
Install as a Cake Addin
#tool nuget:?package=z3l.Scraper&version=0.1.0-preview.6&prerelease
                    
Install as a Cake Tool

z3l.Scraper

Block-aware web scraping engine for .NET 9 and .NET 10.

You write what to take from a page; the scraper takes care of the rest: a request queue with priorities and de-duplication, parallel loading, retries with backoff, recognising blocked, missing and cut-off pages, rotating proxies, rate and time limits, and streaming results while the scraping goes on. HTML is parsed with z3l.FastHtmlParser.

  • Labels and page data. Each request has a label, a PageType (Listing, Item...) or your own string; each label has its handler. A page that knows something about the pages it links to passes it on as typed data.
  • Items of any type. Handlers emit products, reviews, anything; you get them as they come.
  • Response checks. Blocked (403/429, captcha, "Just a moment..."), not found (404, "Product not found"), incomplete (</html> missing, not JSON, a required part missing): configured per label, retried automatically.
  • Retries with backoff and jitter, Retry-After respected, a per-request timeout, a time limit for the whole run. A handler can ask for a retry (RetryRequestException) or give up at once (NonRetryableException).
  • Whole pages or nothing. Items a handler emits reach you when it finishes; if it throws, they are dropped, so a retried page never gives duplicates.
  • Proxies. A proxy list from a text file (http, https, socks4, socks5, with logins), given out in turn; a proxy a site blocks rests for that site, a failing one rests for all.
  • Politeness and limits. A host that answers 429 / 503 or blocks is slowed down by itself while other hosts keep full speed; robots.txt and its Crawl-delay on request; MaxConcurrency, MaxRequestsPerMinute, MaxRequests, MaxDepth, MaxRunDuration.
  • Sitemaps. Sitemap indexes, gzipped and text sitemaps in one call: ctx.EnqueueSitemap(PageType.Item).
  • Runs that survive a crash. With StatePath the queue is saved as it goes and the next run continues; pages in work or not taken yet are done again, nothing is lost.
  • Streaming with backpressure. await foreach over items; a slow consumer slows the scraper down instead of filling the memory.
  • Pluggable loaders. HttpPageLoader is built in; implement IPageLoader for a browser or anything else.
  • Trimming and AOT compatible.

Install

dotnet add package z3l.Scraper --prerelease

Quick start

using z3l.FastHtmlParser;
using z3l.Scraper;
using z3l.Scraper.Loading;

using var loader = new HttpPageLoader();

var scraper = new Scraper(loader)
    .Route(PageType.Listing, ctx =>
    {
        ctx.Enqueue(ctx.Html.XPathValues("//article[@class='product_pod']//h3/a/@href"), PageType.Item);
        ctx.Enqueue(ctx.Html.XPathValue("//li[@class='next']/a/@href"), PageType.Listing);      // next page, null is ignored
    })
    .Route(PageType.Item, ctx => ctx.Emit(new Book(
        Title: ctx.Html.Find(HtmlTag.H1).Text(),
        Price: ctx.Html.Find(HtmlTag.P, cls: "price_color").Text(),
        Url: ctx.Request.Url)));

await foreach (var item in scraper.RunAsync(new ScrapeRequest("https://books.toscrape.com/", PageType.Listing)))
{
    Console.WriteLine(item);
}
Console.WriteLine(scraper.Stats);

record Book(string? Title, string? Price, string Url);

Relative links are resolved against the page (its final URL after redirects); links that are not http(s) (mailto:, javascript:, tel:) are skipped; pages already queued are skipped.

Labels and page data

A label is a PageType or any string; both work everywhere a label is taken (Route, Enqueue, ScrapeRequest, Rules[...]), and labels ignore case. PageType has Root, Category, Listing, Item, ItemDetails, Api, File and Sitemap; File pages are checked as binary and Api pages as JSON unless their rules say otherwise.

A listing needs only its link. A page that already knows something about the pages it links to (a price, a rating, the category) passes it on as data, and the handler of that label gets it typed:

record Teaser(string Price, string Category);

var scraper = new Scraper(loader)
    .Route(PageType.Listing, ctx =>
    {
        foreach (var card in ctx.Html.FindAll(HtmlTag.Div, cls: "card"))
        {
            ctx.Enqueue(card.Find(HtmlTag.A).AttributeHref(), PageType.Item,
                new Teaser(card.Find(HtmlTag.Span, cls: "price").Text()!, "tea"));
        }
        ctx.Enqueue(ctx.Html.XPathValue("//a[@rel='next']/@href"), PageType.Listing);
    })
    .Route<Teaser>(PageType.Item, (ctx, teaser) =>
    {
        ctx.Emit(new Product(ctx.Html.XPathValue("//h1"), teaser.Price, teaser.Category));
        foreach (var review in ctx.Html.FindAll(HtmlTag.Div, cls: "review")) ctx.Emit(new Review(review.Text()));
    })
    .Route("reviews-api", ctx => ...);                      // your own labels next to page types

await foreach (var item in scraper.RunAsync(new ScrapeRequest("https://shop.example/tea", PageType.Listing)))
{
    switch (item)
    {
        case Product product: await db.SaveProductAsync(product); break;
        case Review review: await db.SaveReviewAsync(review); break;
    }
}

Data of the wrong type, or none, for a typed handler is an error where the request is queued, not later. A start request with data: ScrapeRequest.For(url, PageType.Item, new Teaser(...)).

Requests

new ScrapeRequest("https://shop.example/catalog", PageType.Listing);
"https://shop.example/";                                   // a string is a request with the "default" label

new ScrapeRequest("https://shop.example/api/search", PageType.Api)
{
    Method = HttpMethod.Post,
    Body = """{"page":2}""",
    ContentType = "application/json",
    Headers = new Dictionary<string, string> { ["X-Requested-With"] = "XMLHttpRequest" },
    Priority = ScrapePriority.High,
    Data = category,                                       // for Route<Category>, or ctx.Request.Data
    MaxRetries = 10,
};

Start requests need absolute http(s) URLs; a bad one throws ArgumentException at once.

Handlers

scraper.Route(PageType.Item, async ctx =>
{
    ctx.Request;               // the request (Url, Label, Data, Depth, Attempts...)
    ctx.Response;              // StatusCode, Url after redirects, Body, Text, Headers
    ctx.Html;                  // parsed page (z3l.FastHtmlParser), made on first use
    ctx.Json<Product>();       // body as JSON; Json(MyJsonContext.Default.Product) for AOT
    ctx.Enqueue(url, PageType.ItemDetails);
    ctx.Enqueue(url, PageType.ItemDetails, data);
    ctx.Emit(item);            // any type, thread-safe
    await SomethingAsync(ctx.CancellationToken);
});
scraper.FallbackRoute(ctx => ...);    // labels without a handler of their own

A label without a handler is an error at once (InvalidOperationException), not a silent skip.

When a page passes the checks but is still wrong, the handler decides:

scraper.Route(PageType.Item, ctx =>
{
    var price = ctx.Html.Find(HtmlTag.Span, cls: "price").Text();
    if (price is null) throw new RetryRequestException("prices hidden for bots", blocked: true);   // load again, tell the loader
    if (ctx.Html.Find(HtmlTag.Div, cls: "age-gate") != null) throw new NonRetryableException("age gate");   // give up at once
    ctx.Emit(new Product(price));
});

Any other exception fails the request with FailureReason.HandlerError (set Options.RetryOnHandlerError to retry it). Items emitted by a handler that throws are dropped.

A page lives only while its handler runs: then the parser is disposed and the page is let go, even if the consumer is slow to take the items. Items and request data carry values taken from the page (Text(), AttributeHref(), XPathValue()...); Emit and Enqueue refuse tags, the parser and the context, a tag kept anyway throws ObjectDisposedException, and a kept context no longer holds the page. So the scraper holds at most MaxConcurrency pages and ItemBufferSize items; what grows with a run is the set of seen URLs (for duplicates) and the list of failed requests. The text of a page is decoded into a pooled buffer that the response checks, ctx.Html and ctx.Json read in place: a string of it is made only when a handler reads ctx.Response.Text.

Response checks

var options = new ScraperOptions();

// for all labels
options.Rules.Default.Content = ExpectedContent.Html;                 // a cut-off page is retried
options.Rules.Default.BlockedMarkers.AddRange(["Just a moment...", "captcha"]);
options.Rules.Default.BlockedStatusCodes.Add(503);                    // 403 and 429 are there already

// for one label, added to the default rules
options.Rules[PageType.Item].MustContain.Add("itemprop=\"price\"");
options.Rules[PageType.Item].NotFoundMarkers.Add("Product not found");
options.Rules["reviews-api"].Content = ExpectedContent.Json;

// your own check; text is the page read in place (no string), valid during the call
options.Rules[PageType.Item].Custom = (request, response, text) =>
    text.Length < 5000 ? ValidationResult.Retry("page too short") : ValidationResult.Ok;

Checks run in this order; the first that matches decides:

Check Verdict What happens
status in BlockedStatusCodes (403, 429) Blocked the loader is told, the host slows down, retried
a BlockedMarkers string, whatever the status Blocked as above
a NotFoundMarkers string, or status 404 / 410 NotFound retried NotFoundRetries times (1), then failed
any other non-2xx status Retry retried up to MaxRetries (5), then failed
empty body (except 204), wrong format, MustContain, MustNotContain, Custom Retry as above
load error or timeout Retry as above

String checks ignore case. A blocked response or a 429 with Retry-After waits as long as the server asks (up to MaxRetryDelay).

Proxies

var proxies = ProxyList.FromFile("proxies.txt", ProxyBlockPolicy.Block1Hour, logger);
using var loader = new HttpPageLoader(new HttpPageLoaderOptions { Proxies = proxies });

One proxy per line, # for comments:

1.2.3.4:8080                     # http
1.2.3.4:8080:user:password
user:password@1.2.3.4:8080
socks5://user:password@1.2.3.4:1080
https://proxy.example:443

http, https, socks4, socks4a and socks5 proxies, with or without a login, are handled by .NET itself; every proxy has its own connections and cookies. Proxies are given out in turn. When a site blocks an answer, the proxy that loaded it rests for that site only, as ProxyBlockPolicy says: Ignore, Block10Minutes, Block1Hour (the default), Block1Day or Remove.

A failure (no connection, a timeout, 407) or a bad response (the proxy answered, but not with the page: its own page, a cut-off one, the page fails MustContain, the format or another check, comes with a 5xx, or the handler throws RetryRequestException) puts the proxy after the clean ones: it is taken only when no clean proxy is free. Successes do not put a proxy ahead, so the turn stays even and no proxy gets more loads at once than the others. After MaxConsecutiveFailures (3) failures in a row a proxy rests for every site for DeadProxyPause (10 min); after MaxConsecutiveBadResponses (3) bad responses in a row on a site it rests for that site for BadResponsePause (10 min). An error older than its pause is forgotten, a good page starts the count again. Bad responses are counted per site, so a broken site does not take the proxies from the others; 429 and 503 are left to the throttling. When every proxy rests for a site, its loads wait for the first one to come back, no longer than RequestTimeout. The file is read again when it changes; proxies that stay keep their rests. proxies.Status() and proxies.ToString() show what each proxy did.

The list can also come from a proxy service: the URL answers with the same lines or with a JSON array of them, and is read again every 5 minutes in the background (a failed read keeps the old list):

var api = new HttpClient { DefaultRequestHeaders = { { "X-Api-Key", key } } };
var proxies = await ProxyList.FromUrlAsync(new Uri("https://proxies.example/list"), refreshInterval: TimeSpan.FromMinutes(5), httpClient: api);

User agents

using var loader = new HttpPageLoader(new HttpPageLoaderOptions
{
    Proxies = ProxyList.FromFile("proxies.txt"),
    UserAgents = UserAgentList.FromFile("useragents.txt"),     // or await UserAgentList.FromUrlAsync(url)
});

One user agent per line, # for comments; the file is read again when it changes. With proxies every proxy takes one user agent and keeps it, as it keeps its cookies, so a site sees one browser per address; without proxies every request takes the next one. A User-Agent in DefaultHeaders or in the request still wins.

Throttling

A host that answers 429 or 503, or whose responses are blocked (also when a handler throws RetryRequestException(..., blocked: true)), is slowed down as a whole: its requests wait in a line and load one at a time, 1 s apart at first, twice as long after each next such answer (up to MaxThrottleDelay, 1 min), and the delay shrinks again by 10 % with each good page. A Retry-After pauses the host as long as it asks. Requests to other hosts keep going at full speed meanwhile: the line holds requests, not workers. AutoThrottle = false turns it off.

Sitemaps and robots.txt

scraper
    .Route(PageType.Sitemap, ctx => ctx.EnqueueSitemap(PageType.Item,          // pages as Item, nested sitemaps as Sitemap
        filter: page => page.LastModified > DateTimeOffset.UtcNow.AddDays(-7)))
    .Route(PageType.Item, ctx => ...);

await foreach (var item in scraper.RunAsync(new ScrapeRequest("https://shop.example/sitemap.xml", PageType.Sitemap))) { }

ctx.ParseSitemap();                       // Urls (Url, LastModified) and nested Sitemaps, if you want to look first
RobotsTxt.Parse(text).Sitemaps;           // the "Sitemap:" lines of a robots.txt

Sitemaps may be XML (urlset, sitemapindex), gzipped (.xml.gz) or a text list of URLs.

RespectRobotsTxt is off by default; then robots.txt is not even loaded. With RespectRobotsTxt = true the scraper loads robots.txt of each host once, with your loader, before its first page: disallowed pages fail with FailureReason.DisallowedByRobotsTxt without being loaded, and Crawl-delay spaces the loads of the host. Rules are matched as RFC 9309 says (longest match, Allow wins a tie, * and $) for the group of RobotsUserAgent ("z3l-scraper"), or for *. A missing robots.txt allows everything; one that cannot be loaded (5xx, network error) is logged and the host is crawled without rules.

Depth

ctx.Request.Depth is 0 for start requests and one more than the page that queued it. MaxDepth skips deeper links (a page found too deep is still taken if another path reaches it closer).

Resuming a run

var options = new ScraperOptions
{
    StatePath = "state/shop.json",
    StateJson = new DefaultJsonTypeInfoResolver(),        // for typed page data; MyJsonContext.Default for AOT
};

await foreach (var item in scraper.RunAsync(startRequests))
{
    await db.SaveAsync(item);      // an item counts as taken when the loop asks for the next one
}

The state file holds the requests not finished and the keys of the pages seen. It is written every StateSaveInterval (30 s) and when the run stops (time limit, cancellation, break, an exception in your loop), and deleted when the run has finished everything and you have taken every item. Start the program again with the same StatePath and the run continues where it stopped; the start requests are skipped as already seen.

Pages being loaded when the process died, and pages whose items you had not taken yet, are done again: you get every item at least once, so make saving idempotent (an upsert by URL). RunToListAsync keeps items in memory, so a crash loses them; stream with RunAsync when you keep state. Page data must be a plain value (a string, a number, a bool, a Guid, a DateTime(Offset), a JsonElement) or the data of a typed handler, saved as JSON with StateJson: a JsonSerializerContext with your data types (trimming and AOT safe) or new DefaultJsonTypeInfoResolver().

Options

Option Default
MaxConcurrency 8 requests loaded and handled at once
MaxRequestsPerMinute none loads spread evenly, retries included
MaxRequests none accept at most this many requests; handy for trying selectors on a big site
MaxDepth none deepest link to follow; start requests are depth 0
MaxRetries 5 retries after the first attempt
NotFoundRetries 1 retries of a page that looks missing
RetryDelay, RetryBackoffFactor, MaxRetryDelay 1 s, ×2, 1 min delay before retry n is RetryDelay × factorⁿ⁻¹ ±20 %
RequestTimeout 2 min a slower load is cancelled and retried
MaxRunDuration none stop and return what was scraped
AutoThrottle, MaxThrottleDelay on, 1 min slow down a host that answers 429 / 503 or blocks
RespectRobotsTxt, RobotsUserAgent off, "z3l-scraper" skip what robots.txt disallows, honour Crawl-delay
StatePath, StateSaveInterval, StateJson none, 30 s, none save the run and continue it after a crash
RetryOnHandlerError false retry when a handler throws
ItemBufferSize 1000 items waiting for the consumer before workers wait
ProgressInterval 30 s progress line in the log

Results

// stream
await foreach (var item in scraper.RunAsync(requests, cancellationToken)) { ... }

// or collect
var result = await scraper.RunToListAsync(requests);
result.Items;                 // all items, in the order they came
result.ItemsOf<Product>();    // the items of one type
result.Failed;                // FailedRequest: Request, Reason (RetriesExhausted / NotFound / HandlerError / NonRetryable / DisallowedByRobotsTxt), Message, StatusCode, Exception
result.Stats;                 // counters

scraper.RequestFailed += failed => logger.LogWarning("{Url}: {Message}", failed.Request.Url, failed.Message);

scraper.Stats is live during the run: succeeded, failed, pending, queued, in progress, loads, retries, blocked, items, duplicates, elapsed time. Breaking out of await foreach stops the run; the scraper can be run again.

Loaders

// built in: HttpClient with decompression, cookies and connection pooling
using var loader = new HttpPageLoader(new HttpPageLoaderOptions
{
    Proxies = ProxyList.FromFile("proxies.txt"),           // or Proxy = new WebProxy(...) for one; none: the system settings
    UserAgent = "...",                                     // or UserAgents = UserAgentList.FromFile("useragents.txt")
    DefaultHeaders = { ["Accept-Language"] = "de-DE" },
    MaxResponseBytes = 10 * 1024 * 1024,                   // bigger responses fail without retries
});

// or your own client (IHttpClientFactory, Polly...)
var loader = new HttpPageLoader(httpClient);

// or a headless browser: the z3l.Scraper.Playwright package
await using var loader = new PlaywrightPageLoader(new PlaywrightPageLoaderOptions { Channel = "chrome", WaitForSelector = "div.product" });

// or anything else
public sealed class BrowserLoader : IBlockAwarePageLoader
{
    public async Task<PageResponse> LoadAsync(ScrapeRequest request, CancellationToken cancellationToken)
    {
        var html = await ...;
        return PageResponse.FromText(200, finalUrl, html);
    }

    public void ReportBlocked(ScrapeRequest request, PageResponse response, string reason)
    {
        // retire the proxy / browser session that got this response
    }

    // optional: the response was not the page (failed the checks, a 5xx, the handler asked for a retry) / it was the page /
    // the last load of the request ran out of RequestTimeout
    public void ReportBadResponse(ScrapeRequest request, PageResponse response, string reason) { }
    public void ReportGoodResponse(ScrapeRequest request, PageResponse response) { }
    public void ReportTimeout(ScrapeRequest request) { }
}

A loader returns any HTTP status as a normal result and throws only when there is no response; the scraper retries that. A loader throws NonRetryableException when another attempt cannot help. A loader of your own can rotate proxies with ProxyList too: AcquireAsync(host) gives the next proxy that may load from the site, and ReportSuccess, ReportFailure, ReportBlocked, ReportBadResponse and ReportGoodResponse tell the list how it went.

z3l.Scraper.Playwright loads pages in Chromium, Firefox or WebKit, with a browser context per proxy.

Text is decoded the way browsers do it: a byte order mark, the Content-Type charset, then <meta charset> or the XML declaration, then UTF-8. Labels sites use but .NET does not know (cp1251, win-1251, utf8) are understood, and iso-8859-1 is read as windows-1252, so 0x80 is €.

Logging

Pass an ILogger to the constructor: progress, failures and retries (debug level) go there. The scraper writes nothing to the console by itself.

var scraper = new Scraper(loader, options, loggerFactory.CreateLogger("scraper"));

License

Free for companies under USD 1M annual revenue and under 100 employees; others need a commercial license. See LICENSE.md. z3l.FastHtmlParser, which the scraper uses, is licensed separately on the same terms.

Product Compatible and additional computed target framework versions.
.NET net9.0 is compatible.  net9.0-android was computed.  net9.0-browser was computed.  net9.0-ios was computed.  net9.0-maccatalyst was computed.  net9.0-macos was computed.  net9.0-tvos was computed.  net9.0-windows was computed.  net10.0 is compatible.  net10.0-android was computed.  net10.0-browser was computed.  net10.0-ios was computed.  net10.0-maccatalyst was computed.  net10.0-macos was computed.  net10.0-tvos was computed.  net10.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages (1)

Showing the top 1 NuGet packages that depend on z3l.Scraper:

Package Downloads
z3l.Scraper.Playwright

Headless browser page loader for z3l.Scraper on Microsoft Playwright: Chromium, Firefox or WebKit, a browser context per proxy with its own cookies and user agent, rotating proxies and user agents from a file or a URL, blocked images and fonts, waiting for a selector, a session that starts anew after a block.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last Updated
0.1.0-preview.6 0 11/16/2026