They are different companies' crawlers feeding different products. GPTBot and OAI-SearchBot belong to OpenAI, ClaudeBot to Anthropic, and PerplexityBot to Perplexity. Two more names complete the set a site owner should know: Google-Extended, Google's control for AI usage of your content, and Bytespider, ByteDance's crawler. What they share matters more than what separates them: each fetches your page's raw HTML and stops.
These six are exactly the agents our scan checks your robots.txt against, by name.
None of the fetching crawlers above executes JavaScript. Each receives whatever your server returns on the first response, so a page that assembles its content in the browser is blank to all of them at once — which is why fixing rendering fixes your visibility to every assistant simultaneously. The one exception in the ecosystem is Google's Gemini, which reads through Google's own renderer rather than through a separate raw fetch.
You can allow some and block others — robots.txt groups are per-agent, and a crawler obeys the most specific group naming it, not the first permissive line it finds. That precedence rule is the one that catches people out: a User-agent: * allow does not override a later group that names GPTBot with a Disallow. Our free scan parses your robots.txt with the same precedence rules the crawlers use and names exactly who is blocked.
No — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Bytespider all fetch raw HTML and stop. The exception in the wider ecosystem is Google's Gemini, which reads through Google's renderer.
They serve different purposes: GPTBot gathers content for models and products, while OAI-SearchBot supports search-backed answering. They are named separately in robots.txt, so you can treat them differently.
No — it is a robots.txt token Google reads to decide whether your content may be used for AI, not an agent that fetches pages. Blocking it does not remove you from Google Search.
All six named here — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and Bytespider — parsed from your robots.txt with the same group-precedence rules the crawlers themselves use.
Where does your site stand? The scan is free, takes about 60 seconds, and needs no account. It fetches your homepage exactly the way GPTBot, ClaudeBot and PerplexityBot do and tells you what they receive.