How to Allow AI Crawlers

The allow-everything block

Explicit is better than default: stating the allows documents intent and scores full marks on the crawler pillar. Add this to the top of your robots.txt:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

Make sure no later User-agent: * group contains a Disallow: / or aggressive path rules that contradict the allows — more specific groups win, but keep it simple and consistent.

Who each bot is, so you can decide

• GPTBot — OpenAI's crawler for training and corpus-building. Blocking it keeps your content out of ChatGPT's knowledge.
• OAI-SearchBot — powers ChatGPT's search answers. If you want to be cited in ChatGPT search, this one must be allowed.
• ChatGPT-User — fetches pages when a user explicitly asks ChatGPT to open a link.
• ClaudeBot — Anthropic's crawler for Claude.
• PerplexityBot — Perplexity's live crawler; its answers index you in near real time.
• Google-Extended — controls Gemini/AI training use. Blocking it does not affect your Google Search ranking, but does affect AI-surface use.
• CCBot — Common Crawl, a dataset many models train from.

A common middle path: allow the search/browse bots (OAI-SearchBot, ChatGPT-User, PerplexityBot) so you're citable, while deciding per-bot on the training crawlers. Just make it a decision — not a leftover.

Platform notes

• WordPress: most SEO plugins let you edit robots.txt (or create a physical one via FTP). Check the plugin's 'bad bot' blockers — several block AI bots by name.
• Shopify: robots.txt is largely managed for you and by default does not block the major AI bots. You can add rules with robots.txt.liquid if needed.
• Wix/Squarespace/Webflow: all expose robots.txt settings; check them once — builder defaults change without notice.
• Behind Cloudflare or a WAF: bot-fight mode and AI-bot blocking features operate before robots.txt is even read. Server-side blocks override everything above.

Verify it worked

Run your URL through the checker — the crawler pillar shows a per-bot table (explicit allow / default / blocked) in about fifteen seconds. If your site is on an industry template with inherited bot rules, see the per-industry notes: accountants, law firms, ecommerce, home services.