Crumpled.RobotsTxt 3.0.3

Crumpled.RobotsTxt

A flexible, configuration-driven robots.txt solution for Umbraco v13, v14, v15, v16 & v17 that protects your non-production environments from search engine indexing by default, while giving you granular control over crawling rules across multiple sites and environments.

Key Features

• 🛡️ Safe by Default - Blocks all bots by default to prevent accidental indexing of development, staging, or preview environments
• 🌍 Multi-Site & Environment-Aware - Configure different robots.txt rules for different domains/hostnames and environments (Production, Development, Staging, etc.)
• 📝 Flexible Rule Configuration - Define reusable rulesets with Allow/Disallow patterns for different user agents
• 🗺️ Sitemap Integration - Include sitemap URLs per site
• ☁️ Umbraco Cloud Ready - Default behaviour designed for Umbraco Cloud - Perfect for hiding those often overlooked *.umbraco.io environment domains.
• ⚙️ Zero Code Setup - Works out of the box with auto-registration

Install NuGet package

dotnet add package Crumpled.RobotsTxt

Setup

The package automatically registers itself via a Umbraco Composer. No code changes required!

Manual Registration (Advanced)

If you prefer to manually register the package in program.cs, disable the composer:

"Crumpled": {
  "RobotsTxt": {
    "DisableComposer": true
  }
}

Then add to your program.cs:

.AddCrumpledRobotsTxt()

Default Behavior - Protection First

The package prioritizes protecting your content from unintended indexing. When no Sites are configured, smart defaults kick in:

• Custom Default: If you specify a DefaultRuleset, that ruleset will be used as the fallback

• Umbraco Cloud Live Environment: If the environment variable UMBRACO__CLOUD__DEPLOY__ENVIRONMENTNAME equals "live", all bots are allowed by default:

  User-agent: *
  Allow: /
  

• All Other Environments: All bots are blocked by default for safety - protecting staging, development, and preview environments:

  User-agent: *
  Disallow: /
  

⚠️ Note: Once you configure Sites, these defaults are ignored and your custom RuleSets take full control.

Unmatched Domains - Additional Protection

When Sites are configured, any domain that doesn't match the configured HostNames (e.g., temporary preview URLs, forgotten subdomains) will get a protective fallback:

• Custom Default: If you specify a DefaultRuleset, that ruleset will be used
• Otherwise: Blocks all bots for safety:

  User-agent: *
  Disallow: /
  

This prevents unintended crawling of staging, preview, or other unlisted domains - ensuring only your explicitly configured production domains are indexed.

Configuration Example - Multi-Site Setup

Configure different robots.txt rules for different environments and domains using reusable rulesets:

"Crumpled": {
  "RobotsTxt": {
    "DefaultRuleset": "NonProduction",
    "RuleSets": { // There can be multiple rulesets for complex scenarios!
      "Production": {
        "Allow": {
          "*" : ["/"]
        },
        "Disallow": {
          "*": [ "/cdn-cgi/challenge-platform/", "/cdn-cgi/email-platform/" ]
        }
      },
      "NonProduction": { 
        "Allow": {
          "SemrushBot": [ "/" ],
          "SemrushBot-SA": [ "/" ],
          "SemrushBot-Desktop": [ "/" ],
          "SemrushBot-Mobile": [ "/" ],
          "SiteAuditBot": [ "/" ]
        },
        "Disallow": {
          "*": [ "/" ]
        }
      }
    },
    "Sites": {
      "Prod": {
        "HostNames": "www.mysite.com",
        "SiteMapDomain": "www.mysite.com",
        "RuleSet": "Production"
      },
      "AnotherProd": {
        "HostNames": "www.anothermysite.com",
        "SiteMapDomain": "www.anothermysite.com",
        "RuleSet": "Production" // or can define alternate production ruleset for this site
      },
      "Stage": {
        "HostNames": "mysite-staging-uksouth01.umbraco.io,staging.mysite.com", 
        "SiteMapDomain": "staging.mysite.com",
        "RuleSet": "NonProduction"
      },
      "Dev": {
        "HostNames": "mysite-dev-uksouth01.umbraco.io,dev.mysite.com",
        "SiteMapDomain": "dev.mysite.com",
        "RuleSet": "NonProduction"
      }
    }
  }
}