NuGet Gallery | nicholasb.za

RobotsSharpParser

.NET 6.0
30,684 total downloads
last updated 12/22/2021
Latest version: 2.0.0
robots parse robots web crawling robots.txt web scraping spider sitemap More tags

This client library enables working with Robots.txt. Key Features: - Parse robots.txt into Typed object. - Lookup Allowed/Disallowed/Crawldelay based on User-Agent. - Traverse... More information

Nick.RobotsParser

by: nicholasb.za

.NET 8.0
7,436 total downloads
last updated 1/29/2025
Latest version: 2.0.8
robots parse robots web crawling robots.txt web scraping spider sitemap More tags

https://github.com/nicholasbergesen/RobotsParser/blob/master/README.md

Nick.HtmlParser

by: nicholasb.za

.NET 10.0
3,041 total downloads
last updated 2/8/2026
Latest version: 1.0.10
html parser parse html flat structure flatten html

Parses Html into a typed IEnumerable flat structure. v1.0.10 changes: - Bugfix: Improved bounds checking to prevent crashes on malformed HTML (unterminated comments, unclosed tags, lone chevrons). -... More information

RobotsCoreParser

by: nicholasb.za

Deprecated
.NET Core 3.0
1,009 total downloads
last updated 10/23/2019
Latest version: 1.0.0

Used for parsing robots files and sitemaps