That distinction runs through every one of the AI crawlers below, and most robots.txt files I look at get it wrong.
What ClaudeBot actually is, and the two agents next to it
Anthropic's help centre lists three tokens, not one.
ClaudeBot collects web content that could contribute to training. Claude-User fetches a page when a person asks Claude a question about it. Claude-SearchBot crawls to improve search result quality.
All three honour standard robots.txt rules, and Anthropic supports the non-standard Crawl-delay extension, so Crawl-delay: 1 is a real option if crawl rate is your actual complaint.
The catch is that the rules are per-token and per-subdomain. A User-agent: ClaudeBot / Disallow: / block does nothing to Claude-User. Danny Goodwin made the same point in Search Engine Land on 25 February 2026, when Anthropic clarified the three-agent split: you have to write each directive out, on each subdomain.
So a site can be excluded from training and still be read aloud to a user who asks about it. Some publishers want exactly that. Most have not decided, they have copied someone else's robots.txt.
Google-Extended is a robots.txt token, not a bot you can see
This one confuses security teams, because it never shows up in a log.
Google's crawler documentation, last updated 14 July 2026, states plainly that Google-Extended has no separate HTTP user agent string. It is a standalone product token used only for robots.txt control. Googlebot does the fetching, and Google-Extended decides what may be done with what Googlebot already has.
What it controls, per that page: whether content may train future Gemini models, and grounding in Gemini Apps and the Vertex AI API.
What it does not control, in Google's own words, is Search. The page says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."
It also does not remove you from AI Overviews or AI Mode. Google's AI features guidance, last updated 10 December 2025, points site owners to robots.txt directives for Googlebot, plus nosnippet, data-nosnippet, max-snippet and noindex. Google-Extended is a different lever for a different system.
If you searched google extended expecting to find a bot to firewall, that is why you could not find one.
The other AI crawlers worth knowing by name
OpenAI splits the job the same way Anthropic does. Its publishers FAQ says GPTBot is the training crawler, and that a site must allow OAI-SearchBot to be eligible to appear in ChatGPT search. Referral clicks arrive tagged with utm_source=chatgpt.com, which is the cheapest attribution check you will ever run.
Perplexity's documentation says blocking PerplexityBot keeps a site out of Perplexity search results, and that Perplexity-User, the agent that fetches a page because a user asked, generally ignores robots.txt.
Apple's support page, dated 4 September 2026, describes Applebot-Extended the way Google describes Google-Extended: it does not crawl anything itself, it only signals that content should not train Apple's foundation models. Spotlight, Siri and Safari results are unaffected.
Meta documents meta-externalagent for training and indexing and meta-webindexer for Meta AI search quality, both obeying robots.txt, plus meta-externalfetcher, which fetches links at a user's request and, by Meta's own account, may bypass robots.txt.
The pattern repeats across every vendor: a training crawler that respects robots.txt, a search crawler that controls whether you can be cited, and a user-triggered fetcher that often ignores robots.txt entirely.
A large share of that traffic is not the AI company at all
Here is the part that belongs to security rather than marketing. HUMAN Security published an analysis on 9 September 2025 covering 16 to 29 July of that year. It found that 1 in every 18 requests carrying an AI crawler user agent was fake, about 5.7% of traffic labelled as AI crawler, more than 2 million fraudulent requests a day. ChatGPT-User was the most impersonated, at roughly a 1 in 5 spoof ratio. That is vendor research from a bot mitigation company, though the method is documented.
Duane Forrester ran a smaller check and got a much larger number. Writing in Search Engine Journal on 25 June 2026, he verified 14 days of logs on one new site: of 33 requests claiming to be AI assistants, 6 verified, an 81.8% spoof rate, and of 799 requests carrying the Googlebot name, only 107 came from a verified Google address. He says himself to hold the numbers loosely, because six verified is six.
The two studies disagree on magnitude by an order of magnitude, and the sample sizes explain most of that. They agree on the thing that matters: a user agent string is a claim, not an identity, and attackers know that AI crawler names are often allowlisted straight past the WAF.
Cloudflare's investigation on 4 August 2025 found the reverse problem, a real AI company behaving like an attacker. Its researchers set up new domains disallowing all automated access, queried Perplexity about them, and reported that an undeclared crawler posing as Chrome on macOS still retrieved the content, rotating IPs and ASNs outside Perplexity's published range at 3 to 6 million requests a day. Cloudflare de-listed Perplexity as a verified bot, and it is vendor research from a company that sells the blocking product.
How to verify a crawler instead of trusting it
The verification method is old, boring and still correct. Google's documentation, last updated 20 March 2026, describes it: run a reverse DNS lookup on the accessing IP, confirm the hostname is under googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that hostname and confirm it resolves back to the same IP. Google also publishes IP range files, including common-crawlers.json and special-crawlers.json, for matching at scale.
Anthropic publishes a list at claude.com/crawling/bots.json. When I opened it on 17 September 2026 it carried a creation time of 18 August 2026 and 26 IPv4 prefixes, mostly single addresses. Apple publishes applebot.json and uses reverse DNS under applebot.apple.com. OpenAI and Perplexity publish equivalent files.
Published ranges change, so fetch them on a schedule rather than pasting them into a firewall rule once. And Search Engine Land's February 2026 write-up is right that IP blocking alone is unreliable when a vendor runs on shared cloud addresses, so verification is for labelling your logs honestly, not for building your whole defence.
The direction of travel is cryptographic. Cloudflare's Web Bot Auth documentation, last updated 1 July 2026, describes bots signing requests with Ed25519 keys, publishing a key directory at /.well-known/http-message-signatures-directory, and sending Signature-Agent, Signature-Input and Signature headers. It rests on two IETF drafts and is not a finished standard, but it is the first proposal I have seen that makes a bot's identity checkable rather than assertable.
What blocking each one costs you
Blocking a training crawler, ClaudeBot, GPTBot, Google-Extended, Applebot-Extended, meta-externalagent, removes your content from training. It does not remove you from any answer being generated today.
Blocking a search crawler, OAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer, removes you from the retrieval pool those assistants draw on. That is the expensive one, and the one people block by accident when they paste in a blanket AI bot list.
For security vendors the trade is sharper than average, because your buyers ask assistants exactly the questions you want to be present for. In our own 80-query study on 28 August 2026, across 490 cited URLs, comparison pages took 56.7% of citations on head-to-head questions and pricing pages took 53.4% on cost questions. That is unpublished research from my own agency, so treat it as directional rather than settled, but the shape holds: the pages that answer a buying question get cited, and they cannot get cited if the retrieval crawler is disallowed. That is the thinking behind our work on AI search visibility for cybersecurity companies.
None of this is a ranking you can buy. There is no fixed position in ChatGPT and nobody can guarantee placement in an AI answer. What you control is whether your pages are reachable, and whether the thing reaching them is who it says it is.
FAQ
What is ClaudeBot?
ClaudeBot is Anthropic's crawler for web content that may contribute to training Claude. It respects robots.txt and Crawl-delay. It is separate from Claude-User, which fetches pages in response to user questions, and Claude-SearchBot, which crawls for search quality.
Does blocking ClaudeBot stop Claude from mentioning my site?
No. Blocking ClaudeBot addresses training use. Claude-User and Claude-SearchBot are separate tokens and need their own directives, and blocking those is what affects whether your content can be retrieved or indexed for answers.
What is Google Extended, and does blocking it hurt my rankings?
Google-Extended is a robots.txt token with no user agent of its own. It controls training and grounding for Gemini Apps and the Vertex AI API. Google states it does not affect inclusion in Google Search and is not a ranking signal.
How do I tell a real AI crawler from a spoofed one?
Do a reverse DNS lookup on the IP, then a forward lookup on the hostname you get back, and confirm it matches. For vendors that publish IP lists, such as Anthropic's bots.json, Google's crawler JSON files and Apple's applebot.json, match against the current file rather than a cached copy.
Should a security company block AI crawlers by default?
I would block training crawlers only if you have a policy reason to, and I would not block search crawlers without deciding first that you do not want to be cited. The two choices are independent, and most blanket block lists collapse them into one.
I keep finding sites that blocked the retrieval crawler and kept the training crawler open, which is exactly backwards from what they wanted. Have you checked which of the two your robots.txt is actually blocking, or run an audit of what assistants can still reach?
Sources:
Anthropic Help Centre, "Does Anthropic crawl data from the web, and how can site owners block the crawler?", accessed 17 September 2026
Anthropic, claude.com/crawling/bots.json, creation time 18 August 2026, accessed 17 September 2026
Danny Goodwin, Search Engine Land, "Anthropic clarifies how Claude bots crawl sites", 25 February 2026
Google Search Central, "Google crawlers and fetchers", last updated 14 July 2026
Google Search Central, "Verifying Googlebot and other Google crawlers", last updated 20 March 2026
Google Search Central, "AI features and your website", last updated 10 December 2025
Apple Support, "About Applebot", 4 September 2026
Meta for Developers, "Web crawlers", accessed 17 September 2026
OpenAI, Publishers FAQ, accessed 17 September 2026
Perplexity, crawler documentation, accessed 17 September 2026
Aviad Kaiserman, HUMAN Security, "AI crawler spoofing", 9 September 2025, vendor research
Duane Forrester, Search Engine Journal, "81.8% of my AI assistant traffic was fake", 25 June 2026
Gabriel Corral, Vaibhav Singhal, Brian Mitchell and Reid Tatoris, Cloudflare, "Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives", 4 August 2025, vendor research
Cloudflare, "Web Bot Auth", developer documentation, last updated 1 July 2026
GeoCited, 80-query citation study, 28 August 2026, 490 cited URLs, unpublished agency research
About the author: Lourdes Paul Agilan is the founder of GeoCited, an agency that helps B2B companies get recommended in AI search.