Meziantou.Framework.RobotsTxt 2.0.2

Prefix Reserved
.NET 10.0 This package targets .NET 10.0. The package is compatible with this framework or higher. 
dotnet add package Meziantou.Framework.RobotsTxt --version 2.0.2
                    


                    

                
NuGet\Install-Package Meziantou.Framework.RobotsTxt -Version 2.0.2
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package. 
<PackageReference Include="Meziantou.Framework.RobotsTxt" Version="2.0.2" />
For projects that support PackageReference, copy this XML node into the project file to reference the package. 
<PackageVersion Include="Meziantou.Framework.RobotsTxt" Version="2.0.2" />
                    


                            Directory.Packages.props
                        

                    

                
<PackageReference Include="Meziantou.Framework.RobotsTxt" />
                    


                            Project file
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package. 
paket add Meziantou.Framework.RobotsTxt --version 2.0.2
                    


                    

                
#r "nuget: Meziantou.Framework.RobotsTxt, 2.0.2"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package. 
#:package Meziantou.Framework.RobotsTxt@2.0.2
#:package directive can be used in C# file-based apps starting in .NET 10 preview 4. Copy this into a .cs file before any lines of code to reference the package. 
#addin nuget:?package=Meziantou.Framework.RobotsTxt&version=2.0.2
                    


                            Install as a Cake Addin
                        

                    

                
#tool nuget:?package=Meziantou.Framework.RobotsTxt&version=2.0.2
                    


                            Install as a Cake Tool

Meziantou.Framework.RobotsTxt

Meziantou.Framework.RobotsTxt parses robots.txt files (RFC 9309), including Google-style wildcard extensions (* and trailing $).

It supports:

  • User-agent, Allow, Disallow, Crawl-delay, and Sitemap
  • user-agent resolution (exact match preferred over *)
  • path authorization checks
  • lenient parsing with non-fatal parse errors
using Meziantou.Framework.RobotsTxt;

var robots = RobotsFile.Parse("""
User-agent: *
Disallow: /private/
Allow: /private/public/
Crawl-delay: 1.5

Sitemap: https://example.com/sitemap.xml
""");

var allowed = robots.IsAllowed("Googlebot", "/private/public/page"); // true
var blocked = robots.IsAllowed("Googlebot", "/private/secret"); // false
var delay = robots.GetCrawlDelay("Googlebot"); // 00:00:01.5000000
var sitemaps = robots.Sitemaps;

Path rules support wildcards:

var robots = RobotsFile.Parse("""
User-agent: *
Disallow: /*.php$
Allow: /index.php
""");

robots.IsAllowed("AnyBot", "/index.php"); // true
robots.IsAllowed("AnyBot", "/admin.php"); // false

Invalid or unsupported lines do not throw. They are reported in ParseErrors:

var robots = RobotsFile.Parse("""
User-agent: *
Host: example.com
this line has no colon
""");

foreach (var error in robots.ParseErrors)
{
    Console.WriteLine($"{error.LineNumber}: {error.Kind} => {error.Line}");
}
Product Compatible and additional computed target framework versions.
.NET net10.0 is compatible.  net10.0-android was computed.  net10.0-browser was computed.  net10.0-ios was computed.  net10.0-maccatalyst was computed.  net10.0-macos was computed.  net10.0-tvos was computed.  net10.0-windows was computed.  net11.0 is compatible. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

  • net10.0

    • No dependencies.

  • net11.0

    • No dependencies.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last Updated
2.0.2 101 7/17/2026
2.0.1 105 7/8/2026
2.0.0 112 7/5/2026
1.0.0 111 6/24/2026