tsllbn RobotsTxt by: tsllbn 32,951 total downloads last updated 2/19/2014 Latest version: 2014.2.19 Robots.Txt Parser Parsing Library A robots.txt parser for .NET Supports ; - Allow directives. - Crawl-delay directives. - Sitemap declarations. - * and $ wildcards. See https://bitbucket.org/cagdas/robotstxt for usage examples.
RobotsTxt by: tsllbn 32,951 total downloads last updated 2/19/2014 Latest version: 2014.2.19 Robots.Txt Parser Parsing Library A robots.txt parser for .NET Supports ; - Allow directives. - Crawl-delay directives. - Sitemap declarations. - * and $ wildcards. See https://bitbucket.org/cagdas/robotstxt for usage examples.
