Free Robots.txt Generator (2026)

Build a valid robots.txt in seconds. Supports Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, crawl-delay, sitemap, and host directives. No sign-up, no ads.

Rule 1

Global directives

What is robots.txt?

Robots.txt is a plain-text file placed at the root of a domain that follows the Robots Exclusion Protocol (RFC 9309). It tells crawlers like Googlebot, Bingbot, and GPTBot which URLs they may fetch. It does not prevent indexing — use noindex meta tags or HTTP headers for that.

Common bots you can control in 2026

BotOperatorPurpose
GooglebotGoogleWeb search indexing
BingbotMicrosoftBing + Copilot search
GPTBotOpenAIChatGPT training data
OAI-SearchBotOpenAIChatGPT Search citations
ClaudeBotAnthropicClaude training + retrieval
PerplexityBotPerplexityAnswer engine citations
Google-ExtendedGoogleGemini + AI Overviews training
CCBotCommon CrawlOpen dataset used by many LLMs

How to choose your rules

  • Content site or brand: allow all bots including GPTBot, ClaudeBot, and PerplexityBot to maximise AI citations.
  • Paid/proprietary content: block AI training bots (GPTBot, Google-Extended, ClaudeBot, CCBot) but allow search bots.
  • Ecommerce: disallow /cart, /checkout, /account, and internal search URLs.
  • WordPress: disallow /wp-admin/ but allow /wp-admin/admin-ajax.php.

Frequently asked questions

Does robots.txt hide pages from Google?

No. It blocks crawling, not indexing. A blocked URL can still appear in results without a snippet. Use a noindex meta tag to remove pages from the index.

Where do I put the file?

At the domain root: https://yourdomain.com/robots.txt. Subdomains need their own file.

Do I need a sitemap directive?

Yes — add a Sitemap: line so crawlers discover your XML sitemap without extra configuration.

Need help with technical SEO?

I run audits that go beyond robots.txt — crawl budget, indexation, GEO/AI SEO, and growth.

Book a free strategy call