RobotsTxt 2014.2.19
A robots.txt parser for .NET
Supports ;
- Allow directives.
- Crawl-delay directives.
- Sitemap declarations.
- * and $ wildcards.
See https://bitbucket.org/cagdas/robotstxt for usage examples.
Install-Package RobotsTxt -Version 2014.2.19
dotnet add package RobotsTxt --version 2014.2.19
<PackageReference Include="RobotsTxt" Version="2014.2.19" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add RobotsTxt --version 2014.2.19
The NuGet Team does not provide support for this client. Please contact its maintainers for support.
Dependencies
This package has no dependencies.
Used By
NuGet packages
This package is not used by any NuGet packages.
GitHub repositories (1)
Showing the top 1 popular GitHub repositories that depend on RobotsTxt:
| Repository | Stars |
|---|---|
|
esbencarlsen/NCrawler
.NET based webcrawler
|
Version History
| Version | Downloads | Last updated |
|---|---|---|
| 2014.2.19 | 32,034 | 2/19/2014 |
