tsllbn
RobotsTxt : A robots.txt parser for .NET
- 20,356 total downloads
- last updated 2/19/2014
- Latest version: 2014.2.19
A robots.txt parser for .NET
Supports ;
- Allow directives.
- Crawl-delay directives.
- Sitemap declarations.
- * and $ wildcards.
See https://bitbucket.org/cagdas/robotstxt for usage examples.