2 packages returned for Tags:"robotstxt"
-
- 1,615 total downloads
- last updated 4/3/2018
- Latest version: 0.1.0
A tool for accessing and querying the contents of a robots.txt file. -
- 2,969 total downloads
- last updated 12/22/2022
- Latest version: 0.4.0
Parsers for Robots Exclusion Standard (aka robots.txt), Robots Meta Tag, and X-Robots-Tag. Visit Project Site for documentation.