RobotsTxt 2014.2.19

A robots.txt parser for .NET

Supports ;
- Allow directives.
- Crawl-delay directives.
- Sitemap declarations.
- * and $ wildcards.

See https://bitbucket.org/cagdas/robotstxt for usage examples.

Install-Package RobotsTxt -Version 2014.2.19
dotnet add package RobotsTxt --version 2014.2.19
<PackageReference Include="RobotsTxt" Version="2014.2.19" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add RobotsTxt --version 2014.2.19
The NuGet Team does not provide support for this client. Please contact its maintainers for support.

Dependencies

This package has no dependencies.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories (1)

Showing the top 1 popular GitHub repositories that depend on RobotsTxt:

Repository Stars
esbencarlsen/NCrawler
.NET based webcrawler

Version History

Version Downloads Last updated
2014.2.19 32,034 2/19/2014