How do I let GPTBot and ClaudeBot crawl my site?

It is a few lines of robots.txt — and one precedence rule that catches people out.

If you want AI assistants to be able to describe your business, the crawlers have to be allowed to read it. That permission lives in one file at yoursite.com/robots.txt, and the most common mistake is not a missing rule but a rule further down the file quietly overriding a permissive one above it.

The agents worth naming

GPTBot — OpenAI's crawler.
OAI-SearchBot — OpenAI's search crawler, separate from GPTBot.
ClaudeBot — Anthropic's crawler.
PerplexityBot — Perplexity's crawler.
Google-Extended — Google's AI-usage control; it does not affect ordinary Search crawling.
Bytespider — ByteDance's crawler.

The rule that catches people out

A crawler obeys the most specific group that names it, not the first permissive one it finds. So this file blocks ClaudeBot, no matter how welcoming the top looks:

User-agent: *
Allow: /

User-agent: ClaudeBot
Disallow: /

When we report that a site blocks an agent, this precedence is why — we evaluate the group that names the agent, exactly as the crawler does.

A robots.txt that allows them

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Naming each agent with an explicit Allow is redundant against a permissive wildcard, and it is worth doing anyway: it states the intent, and it survives someone later adding a blanket block.

Allowing is not the same as being read

Removing a block lets a crawler in. It does not schedule a visit, and it does not put you in the index an assistant searches. If nothing links to your site, allowing the crawler changes nothing on its own — the crawler still has to find a reason to come.

Straight answers

Where does robots.txt go?

At the root of the domain: https://yoursite.com/robots.txt. It applies to that hostname only, so a subdomain needs its own.

Is no robots.txt the same as blocking?

No — the opposite. If the file is absent, everything is permitted. Absence is permission with the intent simply unrecorded.

Does blocking Google-Extended remove me from Google Search?

No. Google-Extended governs AI usage, not ordinary Search crawling, which follows the Googlebot rules instead.

Do the crawlers obey this?

The named crawlers above publish that they follow robots.txt. It is a convention rather than an enforcement mechanism, so it is not a security control — use real access control for anything private.

Where does your site stand? The scan is free, takes about 60 seconds, and needs no account. It fetches your homepage exactly the way GPTBot, ClaudeBot and PerplexityBot do and tells you what they receive.