z3l.Scraper
0.1.0-preview.6
dotnet add package z3l.Scraper --version 0.1.0-preview.6
NuGet\Install-Package z3l.Scraper -Version 0.1.0-preview.6
<PackageReference Include="z3l.Scraper" Version="0.1.0-preview.6" />
<PackageVersion Include="z3l.Scraper" Version="0.1.0-preview.6" />
<PackageReference Include="z3l.Scraper" />
paket add z3l.Scraper --version 0.1.0-preview.6
#r "nuget: z3l.Scraper, 0.1.0-preview.6"
#:package z3l.Scraper@0.1.0-preview.6
#addin nuget:?package=z3l.Scraper&version=0.1.0-preview.6&prerelease
#tool nuget:?package=z3l.Scraper&version=0.1.0-preview.6&prerelease
z3l.Scraper
Block-aware web scraping engine for .NET 9 and .NET 10.
You write what to take from a page; the scraper takes care of the rest: a request queue with priorities and de-duplication, parallel loading, retries with backoff, recognising blocked, missing and cut-off pages, rotating proxies, rate and time limits, and streaming results while the scraping goes on. HTML is parsed with z3l.FastHtmlParser.
- Labels and page data. Each request has a label, a
PageType(Listing,Item...) or your own string; each label has its handler. A page that knows something about the pages it links to passes it on as typed data. - Items of any type. Handlers emit products, reviews, anything; you get them as they come.
- Response checks. Blocked (403/429, captcha, "Just a moment..."), not found (404, "Product not found"),
incomplete (
</html>missing, not JSON, a required part missing): configured per label, retried automatically. - Retries with backoff and jitter,
Retry-Afterrespected, a per-request timeout, a time limit for the whole run. A handler can ask for a retry (RetryRequestException) or give up at once (NonRetryableException). - Whole pages or nothing. Items a handler emits reach you when it finishes; if it throws, they are dropped, so a retried page never gives duplicates.
- Proxies. A proxy list from a text file (http, https, socks4, socks5, with logins), given out in turn; a proxy a site blocks rests for that site, a failing one rests for all.
- Politeness and limits. A host that answers 429 / 503 or blocks is slowed down by itself while other hosts keep
full speed; robots.txt and its Crawl-delay on request;
MaxConcurrency,MaxRequestsPerMinute,MaxRequests,MaxDepth,MaxRunDuration. - Sitemaps. Sitemap indexes, gzipped and text sitemaps in one call:
ctx.EnqueueSitemap(PageType.Item). - Runs that survive a crash. With
StatePaththe queue is saved as it goes and the next run continues; pages in work or not taken yet are done again, nothing is lost. - Streaming with backpressure.
await foreachover items; a slow consumer slows the scraper down instead of filling the memory. - Pluggable loaders.
HttpPageLoaderis built in; implementIPageLoaderfor a browser or anything else. - Trimming and AOT compatible.
Install
dotnet add package z3l.Scraper --prerelease
Quick start
using z3l.FastHtmlParser;
using z3l.Scraper;
using z3l.Scraper.Loading;
using var loader = new HttpPageLoader();
var scraper = new Scraper(loader)
.Route(PageType.Listing, ctx =>
{
ctx.Enqueue(ctx.Html.XPathValues("//article[@class='product_pod']//h3/a/@href"), PageType.Item);
ctx.Enqueue(ctx.Html.XPathValue("//li[@class='next']/a/@href"), PageType.Listing); // next page, null is ignored
})
.Route(PageType.Item, ctx => ctx.Emit(new Book(
Title: ctx.Html.Find(HtmlTag.H1).Text(),
Price: ctx.Html.Find(HtmlTag.P, cls: "price_color").Text(),
Url: ctx.Request.Url)));
await foreach (var item in scraper.RunAsync(new ScrapeRequest("https://books.toscrape.com/", PageType.Listing)))
{
Console.WriteLine(item);
}
Console.WriteLine(scraper.Stats);
record Book(string? Title, string? Price, string Url);
Relative links are resolved against the page (its final URL after redirects); links that are not http(s)
(mailto:, javascript:, tel:) are skipped; pages already queued are skipped.
Labels and page data
A label is a PageType or any string; both work everywhere a label is taken (Route, Enqueue, ScrapeRequest,
Rules[...]), and labels ignore case. PageType has Root, Category, Listing, Item, ItemDetails, Api, File
and Sitemap; File pages are checked as binary and Api pages as JSON unless their rules say otherwise.
A listing needs only its link. A page that already knows something about the pages it links to (a price, a rating, the category) passes it on as data, and the handler of that label gets it typed:
record Teaser(string Price, string Category);
var scraper = new Scraper(loader)
.Route(PageType.Listing, ctx =>
{
foreach (var card in ctx.Html.FindAll(HtmlTag.Div, cls: "card"))
{
ctx.Enqueue(card.Find(HtmlTag.A).AttributeHref(), PageType.Item,
new Teaser(card.Find(HtmlTag.Span, cls: "price").Text()!, "tea"));
}
ctx.Enqueue(ctx.Html.XPathValue("//a[@rel='next']/@href"), PageType.Listing);
})
.Route<Teaser>(PageType.Item, (ctx, teaser) =>
{
ctx.Emit(new Product(ctx.Html.XPathValue("//h1"), teaser.Price, teaser.Category));
foreach (var review in ctx.Html.FindAll(HtmlTag.Div, cls: "review")) ctx.Emit(new Review(review.Text()));
})
.Route("reviews-api", ctx => ...); // your own labels next to page types
await foreach (var item in scraper.RunAsync(new ScrapeRequest("https://shop.example/tea", PageType.Listing)))
{
switch (item)
{
case Product product: await db.SaveProductAsync(product); break;
case Review review: await db.SaveReviewAsync(review); break;
}
}
Data of the wrong type, or none, for a typed handler is an error where the request is queued, not later.
A start request with data: ScrapeRequest.For(url, PageType.Item, new Teaser(...)).
Requests
new ScrapeRequest("https://shop.example/catalog", PageType.Listing);
"https://shop.example/"; // a string is a request with the "default" label
new ScrapeRequest("https://shop.example/api/search", PageType.Api)
{
Method = HttpMethod.Post,
Body = """{"page":2}""",
ContentType = "application/json",
Headers = new Dictionary<string, string> { ["X-Requested-With"] = "XMLHttpRequest" },
Priority = ScrapePriority.High,
Data = category, // for Route<Category>, or ctx.Request.Data
MaxRetries = 10,
};
Start requests need absolute http(s) URLs; a bad one throws ArgumentException at once.
Handlers
scraper.Route(PageType.Item, async ctx =>
{
ctx.Request; // the request (Url, Label, Data, Depth, Attempts...)
ctx.Response; // StatusCode, Url after redirects, Body, Text, Headers
ctx.Html; // parsed page (z3l.FastHtmlParser), made on first use
ctx.Json<Product>(); // body as JSON; Json(MyJsonContext.Default.Product) for AOT
ctx.Enqueue(url, PageType.ItemDetails);
ctx.Enqueue(url, PageType.ItemDetails, data);
ctx.Emit(item); // any type, thread-safe
await SomethingAsync(ctx.CancellationToken);
});
scraper.FallbackRoute(ctx => ...); // labels without a handler of their own
A label without a handler is an error at once (InvalidOperationException), not a silent skip.
When a page passes the checks but is still wrong, the handler decides:
scraper.Route(PageType.Item, ctx =>
{
var price = ctx.Html.Find(HtmlTag.Span, cls: "price").Text();
if (price is null) throw new RetryRequestException("prices hidden for bots", blocked: true); // load again, tell the loader
if (ctx.Html.Find(HtmlTag.Div, cls: "age-gate") != null) throw new NonRetryableException("age gate"); // give up at once
ctx.Emit(new Product(price));
});
Any other exception fails the request with FailureReason.HandlerError (set Options.RetryOnHandlerError to retry it).
Items emitted by a handler that throws are dropped.
A page lives only while its handler runs: then the parser is disposed and the page is let go, even if the consumer is
slow to take the items. Items and request data carry values taken from the page (Text(), AttributeHref(),
XPathValue()...); Emit and Enqueue refuse tags, the parser and the context, a tag kept anyway throws
ObjectDisposedException, and a kept context no longer holds the page. So the scraper holds at most MaxConcurrency
pages and ItemBufferSize items; what grows with a run is the set of seen URLs (for duplicates) and the list of
failed requests. The text of a page is decoded into a pooled buffer that the response checks, ctx.Html and ctx.Json
read in place: a string of it is made only when a handler reads ctx.Response.Text.
Response checks
var options = new ScraperOptions();
// for all labels
options.Rules.Default.Content = ExpectedContent.Html; // a cut-off page is retried
options.Rules.Default.BlockedMarkers.AddRange(["Just a moment...", "captcha"]);
options.Rules.Default.BlockedStatusCodes.Add(503); // 403 and 429 are there already
// for one label, added to the default rules
options.Rules[PageType.Item].MustContain.Add("itemprop=\"price\"");
options.Rules[PageType.Item].NotFoundMarkers.Add("Product not found");
options.Rules["reviews-api"].Content = ExpectedContent.Json;
// your own check; text is the page read in place (no string), valid during the call
options.Rules[PageType.Item].Custom = (request, response, text) =>
text.Length < 5000 ? ValidationResult.Retry("page too short") : ValidationResult.Ok;
Checks run in this order; the first that matches decides:
| Check | Verdict | What happens |
|---|---|---|
status in BlockedStatusCodes (403, 429) |
Blocked | the loader is told, the host slows down, retried |
a BlockedMarkers string, whatever the status |
Blocked | as above |
a NotFoundMarkers string, or status 404 / 410 |
NotFound | retried NotFoundRetries times (1), then failed |
| any other non-2xx status | Retry | retried up to MaxRetries (5), then failed |
empty body (except 204), wrong format, MustContain, MustNotContain, Custom |
Retry | as above |
| load error or timeout | Retry | as above |
String checks ignore case. A blocked response or a 429 with Retry-After waits as long as the server asks
(up to MaxRetryDelay).
Proxies
var proxies = ProxyList.FromFile("proxies.txt", ProxyBlockPolicy.Block1Hour, logger);
using var loader = new HttpPageLoader(new HttpPageLoaderOptions { Proxies = proxies });
One proxy per line, # for comments:
1.2.3.4:8080 # http
1.2.3.4:8080:user:password
user:password@1.2.3.4:8080
socks5://user:password@1.2.3.4:1080
https://proxy.example:443
http, https, socks4, socks4a and socks5 proxies, with or without a login, are handled by .NET itself; every proxy has
its own connections and cookies. Proxies are given out in turn. When a site blocks an answer, the proxy that loaded it
rests for that site only, as ProxyBlockPolicy says: Ignore, Block10Minutes, Block1Hour (the default),
Block1Day or Remove.
A failure (no connection, a timeout, 407) or a bad response (the proxy answered, but not with the page: its own page, a
cut-off one, the page fails MustContain, the format or another check, comes with a 5xx, or the handler throws
RetryRequestException) puts the proxy after the clean ones: it is taken only when no clean proxy is free. Successes do
not put a proxy ahead, so the turn stays even and no proxy gets more loads at once than the others. After
MaxConsecutiveFailures (3) failures in a row a proxy rests for every site for DeadProxyPause (10 min); after
MaxConsecutiveBadResponses (3) bad responses in a row on a site it rests for that site for BadResponsePause (10 min).
An error older than its pause is forgotten, a good page starts the count again. Bad responses are counted per site, so a
broken site does not take the proxies from the others; 429 and 503 are left to the throttling. When every proxy rests for a site, its loads wait for the first one to come back,
no longer than RequestTimeout. The file is read again when it changes; proxies that stay keep their rests.
proxies.Status() and proxies.ToString() show what each proxy did.
The list can also come from a proxy service: the URL answers with the same lines or with a JSON array of them, and is read again every 5 minutes in the background (a failed read keeps the old list):
var api = new HttpClient { DefaultRequestHeaders = { { "X-Api-Key", key } } };
var proxies = await ProxyList.FromUrlAsync(new Uri("https://proxies.example/list"), refreshInterval: TimeSpan.FromMinutes(5), httpClient: api);
User agents
using var loader = new HttpPageLoader(new HttpPageLoaderOptions
{
Proxies = ProxyList.FromFile("proxies.txt"),
UserAgents = UserAgentList.FromFile("useragents.txt"), // or await UserAgentList.FromUrlAsync(url)
});
One user agent per line, # for comments; the file is read again when it changes. With proxies every proxy takes one
user agent and keeps it, as it keeps its cookies, so a site sees one browser per address; without proxies every request
takes the next one. A User-Agent in DefaultHeaders or in the request still wins.
Throttling
A host that answers 429 or 503, or whose responses are blocked (also when a handler throws
RetryRequestException(..., blocked: true)), is slowed down as a whole: its requests wait in a line and load one at a time,
1 s apart at first, twice as long after each next such answer (up to MaxThrottleDelay, 1 min), and the delay shrinks
again by 10 % with each good page. A Retry-After pauses the host as long as it asks. Requests to other hosts keep
going at full speed meanwhile: the line holds requests, not workers. AutoThrottle = false turns it off.
Sitemaps and robots.txt
scraper
.Route(PageType.Sitemap, ctx => ctx.EnqueueSitemap(PageType.Item, // pages as Item, nested sitemaps as Sitemap
filter: page => page.LastModified > DateTimeOffset.UtcNow.AddDays(-7)))
.Route(PageType.Item, ctx => ...);
await foreach (var item in scraper.RunAsync(new ScrapeRequest("https://shop.example/sitemap.xml", PageType.Sitemap))) { }
ctx.ParseSitemap(); // Urls (Url, LastModified) and nested Sitemaps, if you want to look first
RobotsTxt.Parse(text).Sitemaps; // the "Sitemap:" lines of a robots.txt
Sitemaps may be XML (urlset, sitemapindex), gzipped (.xml.gz) or a text list of URLs.
RespectRobotsTxt is off by default; then robots.txt is not even loaded. With RespectRobotsTxt = true the scraper loads
robots.txt of each host once, with your loader, before its first page: disallowed pages fail with
FailureReason.DisallowedByRobotsTxt without being loaded, and Crawl-delay spaces the loads of the host. Rules are matched
as RFC 9309 says (longest match, Allow wins a tie, * and $) for the group of RobotsUserAgent ("z3l-scraper"), or for
*. A missing robots.txt allows everything; one that cannot be loaded (5xx, network error) is logged and the host is
crawled without rules.
Depth
ctx.Request.Depth is 0 for start requests and one more than the page that queued it. MaxDepth skips deeper links
(a page found too deep is still taken if another path reaches it closer).
Resuming a run
var options = new ScraperOptions
{
StatePath = "state/shop.json",
StateJson = new DefaultJsonTypeInfoResolver(), // for typed page data; MyJsonContext.Default for AOT
};
await foreach (var item in scraper.RunAsync(startRequests))
{
await db.SaveAsync(item); // an item counts as taken when the loop asks for the next one
}
The state file holds the requests not finished and the keys of the pages seen. It is written every
StateSaveInterval (30 s) and when the run stops (time limit, cancellation, break, an exception in your loop), and
deleted when the run has finished everything and you have taken every item. Start the program again with the same
StatePath and the run continues where it stopped; the start requests are skipped as already seen.
Pages being loaded when the process died, and pages whose items you had not taken yet, are done again: you get every
item at least once, so make saving idempotent (an upsert by URL). RunToListAsync keeps items in memory, so a crash
loses them; stream with RunAsync when you keep state. Page data must be a plain value (a string, a number, a bool,
a Guid, a DateTime(Offset), a JsonElement) or the data of a typed handler, saved as JSON with StateJson: a
JsonSerializerContext with your data types (trimming and AOT safe) or new DefaultJsonTypeInfoResolver().
Options
| Option | Default | |
|---|---|---|
MaxConcurrency |
8 | requests loaded and handled at once |
MaxRequestsPerMinute |
none | loads spread evenly, retries included |
MaxRequests |
none | accept at most this many requests; handy for trying selectors on a big site |
MaxDepth |
none | deepest link to follow; start requests are depth 0 |
MaxRetries |
5 | retries after the first attempt |
NotFoundRetries |
1 | retries of a page that looks missing |
RetryDelay, RetryBackoffFactor, MaxRetryDelay |
1 s, ×2, 1 min | delay before retry n is RetryDelay × factorⁿ⁻¹ ±20 % |
RequestTimeout |
2 min | a slower load is cancelled and retried |
MaxRunDuration |
none | stop and return what was scraped |
AutoThrottle, MaxThrottleDelay |
on, 1 min | slow down a host that answers 429 / 503 or blocks |
RespectRobotsTxt, RobotsUserAgent |
off, "z3l-scraper" | skip what robots.txt disallows, honour Crawl-delay |
StatePath, StateSaveInterval, StateJson |
none, 30 s, none | save the run and continue it after a crash |
RetryOnHandlerError |
false | retry when a handler throws |
ItemBufferSize |
1000 | items waiting for the consumer before workers wait |
ProgressInterval |
30 s | progress line in the log |
Results
// stream
await foreach (var item in scraper.RunAsync(requests, cancellationToken)) { ... }
// or collect
var result = await scraper.RunToListAsync(requests);
result.Items; // all items, in the order they came
result.ItemsOf<Product>(); // the items of one type
result.Failed; // FailedRequest: Request, Reason (RetriesExhausted / NotFound / HandlerError / NonRetryable / DisallowedByRobotsTxt), Message, StatusCode, Exception
result.Stats; // counters
scraper.RequestFailed += failed => logger.LogWarning("{Url}: {Message}", failed.Request.Url, failed.Message);
scraper.Stats is live during the run: succeeded, failed, pending, queued, in progress, loads, retries, blocked,
items, duplicates, elapsed time. Breaking out of await foreach stops the run; the scraper can be run again.
Loaders
// built in: HttpClient with decompression, cookies and connection pooling
using var loader = new HttpPageLoader(new HttpPageLoaderOptions
{
Proxies = ProxyList.FromFile("proxies.txt"), // or Proxy = new WebProxy(...) for one; none: the system settings
UserAgent = "...", // or UserAgents = UserAgentList.FromFile("useragents.txt")
DefaultHeaders = { ["Accept-Language"] = "de-DE" },
MaxResponseBytes = 10 * 1024 * 1024, // bigger responses fail without retries
});
// or your own client (IHttpClientFactory, Polly...)
var loader = new HttpPageLoader(httpClient);
// or a headless browser: the z3l.Scraper.Playwright package
await using var loader = new PlaywrightPageLoader(new PlaywrightPageLoaderOptions { Channel = "chrome", WaitForSelector = "div.product" });
// or anything else
public sealed class BrowserLoader : IBlockAwarePageLoader
{
public async Task<PageResponse> LoadAsync(ScrapeRequest request, CancellationToken cancellationToken)
{
var html = await ...;
return PageResponse.FromText(200, finalUrl, html);
}
public void ReportBlocked(ScrapeRequest request, PageResponse response, string reason)
{
// retire the proxy / browser session that got this response
}
// optional: the response was not the page (failed the checks, a 5xx, the handler asked for a retry) / it was the page /
// the last load of the request ran out of RequestTimeout
public void ReportBadResponse(ScrapeRequest request, PageResponse response, string reason) { }
public void ReportGoodResponse(ScrapeRequest request, PageResponse response) { }
public void ReportTimeout(ScrapeRequest request) { }
}
A loader returns any HTTP status as a normal result and throws only when there is no response; the scraper
retries that. A loader throws NonRetryableException when another attempt cannot help. A loader of your own can
rotate proxies with ProxyList too: AcquireAsync(host) gives the next proxy that may load from the site, and
ReportSuccess, ReportFailure, ReportBlocked, ReportBadResponse and ReportGoodResponse tell the list how it went.
z3l.Scraper.Playwright loads pages in Chromium, Firefox or WebKit, with a browser context per proxy.
Text is decoded the way browsers do it: a byte order mark, the Content-Type charset, then <meta charset> or the
XML declaration, then UTF-8. Labels sites use but .NET does not know (cp1251, win-1251, utf8) are understood,
and iso-8859-1 is read as windows-1252, so 0x80 is €.
Logging
Pass an ILogger to the constructor: progress, failures and retries (debug level) go there. The scraper writes
nothing to the console by itself.
var scraper = new Scraper(loader, options, loggerFactory.CreateLogger("scraper"));
License
Free for companies under USD 1M annual revenue and under 100 employees; others need a commercial license. See LICENSE.md. z3l.FastHtmlParser, which the scraper uses, is licensed separately on the same terms.
| Product | Versions Compatible and additional computed target framework versions. |
|---|---|
| .NET | net9.0 is compatible. net9.0-android was computed. net9.0-browser was computed. net9.0-ios was computed. net9.0-maccatalyst was computed. net9.0-macos was computed. net9.0-tvos was computed. net9.0-windows was computed. net10.0 is compatible. net10.0-android was computed. net10.0-browser was computed. net10.0-ios was computed. net10.0-maccatalyst was computed. net10.0-macos was computed. net10.0-tvos was computed. net10.0-windows was computed. |
-
net10.0
- Microsoft.Extensions.Logging.Abstractions (>= 9.0.0)
- z3l.FastHtmlParser (>= 1.4.0)
-
net9.0
- Microsoft.Extensions.Logging.Abstractions (>= 9.0.0)
- z3l.FastHtmlParser (>= 1.4.0)
NuGet packages (1)
Showing the top 1 NuGet packages that depend on z3l.Scraper:
| Package | Downloads |
|---|---|
|
z3l.Scraper.Playwright
Headless browser page loader for z3l.Scraper on Microsoft Playwright: Chromium, Firefox or WebKit, a browser context per proxy with its own cookies and user agent, rotating proxies and user agents from a file or a URL, blocked images and fonts, waiting for a selector, a session that starts anew after a block. |
GitHub repositories
This package is not used by any popular GitHub repositories.
| Version | Downloads | Last Updated |
|---|---|---|
| 0.1.0-preview.6 | 0 | 11/16/2026 |