Robots.AspNet 1.2.0

Robots.AspNet

A lightweight ASP.NET Core library that serves a dynamic robots.txt file via middleware, sets X-Robots-Tag headers on matching paths, with a fluent builder API and appsettings.json support.

Quick start

Install:

dotnet add package Robots.AspNet

In Program.cs:

using Robots.AspNet;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddRobotsTxt()
    .Disallow("GPTBot", "/")
    .Disallow("*", "/admin/", "/login/")
    .Allow("/login/blabla")
    .CrawlDelay(10)
    .Sitemap("https://example.com/sitemap.xml")
    .AddRobotsTag("/admin/*", "noindex, nofollow")
    .Comment("Generated by Robots.AspNet");

var app = builder.Build();

app.UseRobotsTxt();
app.Run();

API

AddRobotsTxt() — returns IRobotsTxtBuilder

services.AddRobotsTxt()
services.AddRobotsTxt(IConfiguration)
services.AddRobotsTxt(Action<RobotsTxtOptions>)

IRobotsTxtBuilder fluent methods:

• Allow(path) and CrawlDelay(seconds) without a user-agent apply to the user-agent from the preceding Disallow() call
• Duplicate paths and sitemaps are silently ignored

AddRobotsTag(path, value) — X-Robots-Tag headers

Controls how crawlers index specific paths by setting the X-Robots-Tag response header.

Path patterns: | Pattern | Example | Matches | |---|---|---| | * | AddRobotsTag("*", "noindex") | All paths | | /admin/* | AddRobotsTag("/admin/*", "none") | /admin/, /admin/settings, etc. | | /secret | AddRobotsTag("/secret", "noindex") | Only that exact path |

The first matching rule wins. The header is not applied to /robots.txt.

services.AddRobotsTxt()
    .AddRobotsTag("/admin/*", "noindex, nofollow")
    .AddRobotsTag("/private/", "noindex");

Middleware

app.UseRobotsTxt();

• /robots.txt — returns the generated content as text/plain; charset=utf-8 (GET/HEAD only, other methods return 405). Sets Content-Length and optional Cache-Control.
• Other paths — if any AddRobotsTag rule matches the request path, the X-Robots-Tag header is set on the response before passing through.

appsettings.json

{
  "RobotsTxt": {
    "Rules": [
      {
        "UserAgent": "GPTBot",
        "Disallow": ["/"]
      },
      {
        "UserAgent": "*",
        "Disallow": ["/admin/", "/login/"],
        "Allow": ["/login/blabla"],
        "CrawlDelay": 10
      }
    ],
    "Sitemaps": ["https://example.com/sitemap.xml"],
    "Comments": ["Managed by Robots.AspNet"],
    "Host": "https://www.example.com",
    "CacheMaxAgeSeconds": 86400,
    "Tags": [
      { "Path": "/admin/*", "Value": "noindex, nofollow" },
      { "Path": "/private/", "Value": "noindex" }
    ]
  }
}

builder.Services.AddRobotsTxt(builder.Configuration.GetSection("RobotsTxt"))
    .Disallow("BadBot", "/");

Output format

Follows RFC 9309 — Robots Exclusion Protocol:

• UTF-8 encoded, text/plain content type
• Groups separated by blank lines
• Rules within a group are output in registration order

NuGet

dotnet pack src/Robots.AspNet

Requires no NuGet dependencies beyond the ASP.NET Core shared framework. Package includes symbols (snupkg).

Project structure

Robots.AspNet/
├── README.md
├── Robots.AspNet.slnx
└── src/
    ├── Robots.AspNet/               # Library
    │   ├── IRobotsTxtBuilder.cs
    │   ├── RobotsTxtBuilder.cs
    │   ├── RobotsTxtMiddleware.cs
    │   ├── RobotsTxtOptions.cs
    │   ├── RobotsTxtRule.cs
    │   ├── RobotsTagRule.cs
    │   ├── ServiceCollectionExtensions.cs
    │   └── ApplicationBuilderExtensions.cs
    └── Robots.AspNet.Tests/         # NUnit tests
        ├── RobotsTxtBuilderTests.cs
        ├── RobotsTxtMiddlewareTests.cs
        └── ConfigurationBindingTests.cs