Which AI crawlers should I allow?
Allow the search crawlers, because they are the only ones that produce citations with a link: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot and Applebot. Decide separately about the training crawlers GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot, because they do not feed your visibility today, they feed the model knowledge of future generations. Leave the user-initiated fetches such as ChatGPT-User or Perplexity-User open: there is a real person behind each of them who just asked about you.
- Three purposes, three separate decisions: search, training, user-initiated fetch. Only search produces citations with a link.
- GPTBot and OAI-SearchBot are two different bots. Blocking both removes you from ChatGPT answers without you meaning to (OpenAI docs 2026).
- Google-Extended and Applebot-Extended govern training only. Blocking both costs you no visibility in search (Google Search Central, Apple Support 2026).
- Training accounted for nearly 80 percent of AI crawler traffic in July 2025, user actions and undeclared purposes together for under 5 percent (Cloudflare, August 28, 2025).
- A bot reads only the group that matches its name. Once its name appears in the file, it ignores
User-agent: *entirely.
What are the providers crawling for?
For three different purposes, and the major providers now separate those purposes by bot themselves: search, training and user-initiated fetching. Write a single rule for all AI bots and you make three decisions at once without ever making them.
- Search: the bot builds an index that later produces answers with citations. This is the only purpose that earns you citations and links.
- Training: the bot collects material for future model generations. Whatever it takes appears with the next training run at the earliest, without a link and without naming your domain.
- User-initiated fetch: somebody pasted a link into a chat or asked about your offer, and the system pulls the page live. That is not crawling in the usual sense, it is a single request with a person behind it.
Cloudflare has measured how unevenly those three purposes are distributed: in July 2025, nearly 80 percent of AI bot crawling went to training, while user actions together with undeclared purposes stayed under 5 percent (Cloudflare, August 28, 2025). The purpose that creates the least load is the same one most likely to send someone your way.
Which bot serves which purpose?
This table is the actual decision. It lists the common bots with the purpose the provider states, plus the recommendation for a site that wins customers through visibility. As of August 2026, checked against the provider documentation.
| Bot | Stated purpose | robots.txt | Recommendation |
|---|---|---|---|
OAI-SearchBot | OpenAI: search features in ChatGPT | obeyed | allow |
GPTBot | OpenAI: training the foundation models | obeyed | your call |
ChatGPT-User | OpenAI: user-initiated fetch | may not apply | allow |
OAI-AdsBot | OpenAI: safety checks on pages submitted as ads | obeyed | only if you advertise |
Claude-SearchBot | Anthropic: quality of search results | obeyed | allow |
ClaudeBot | Anthropic: content that may contribute to training | obeyed | your call |
Claude-User | Anthropic: user-initiated fetch | obeyed | allow |
PerplexityBot | Perplexity: search results, not used for training | obeyed | allow |
Perplexity-User | Perplexity: user-initiated fetch | generally ignored | allow |
Googlebot | Google: search index, the basis for AI Overviews and AI Mode | obeyed | allow |
Google-Extended | Google: training and grounding for Gemini and Vertex AI | obeyed | your call |
Google-CloudVertexBot | Google: Vertex AI agents at the site owner's request | obeyed | only if you use it |
Bingbot | Microsoft: search index, the basis for Copilot | obeyed | allow |
Applebot | Apple: search in Spotlight, Siri and Safari | obeyed | allow |
Applebot-Extended | Apple: training the foundation models | obeyed | your call |
meta-webindexer | Meta: search results in Meta AI | obeyed | allow |
meta-externalagent | Meta: training and indexing for products | obeyed | your call |
meta-externalfetcher | Meta: user-initiated fetch | may be bypassed | allow |
CCBot | Common Crawl: open dataset used by many models | obeyed | your call |
Two rows deserve extra attention, because they are the ones most often misread. Google-Extended is not a crawler, it is a switch: it governs whether content Google has already crawled may be used to train future Gemini models, and it has no effect on inclusion in search (Google Search Central, 2026). Applebot-Extended works on the same principle, and Apple spells it out: the token does not crawl pages at all, and pages that disallow it can still appear in search results (Apple Support, 2026). They are the only blocks on this list that are guaranteed to cost you nothing.
What does a blanket block cost you?
Visibility in exactly the answers that link out. Block every AI bot and you disappear from ChatGPT, Perplexity and Copilot answers, without recovering the training data that already sits in older model runs. The protection works forwards, the loss lands immediately.
The other side still has good arguments, and they can be quantified. Cloudflare compares crawling against the visits sent back: in early August 2025, Anthropic reached a ratio of nearly 50,000 pages crawled per referral, OpenAI 887 to 1 and Perplexity 118 to 1 (Cloudflare, August 28, 2025). For an ad-funded publication that sells every page view, that is a bad deal. For a company that wins customers through visibility, the maths differs: there the mention in the answer is the value, click or no click.
So the honest dividing line does not run between open and closed, it runs between the three purposes. For most company sites the answer is: allow search, allow user fetches, and decide on training according to how you feel about the question.
What does a robots.txt that separates them look like?
One rule first, the one most hand-built files get wrong: a bot reads only the group that matches its name. Once its name appears anywhere in the file, it ignores User-agent: * entirely, including everything that group would have allowed. Name individual bots and you have to repeat every rule for them instead of relying on the catch-all.
For a company site that sells through visibility, the open profile is enough:
User-agent: *
Allow: /
Sitemap: https://your-domain.com/sitemap.xmlTo allow search while limiting training, name both sides explicitly:
# Search and answers: welcome
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: Applebot
Allow: /
User-agent: meta-webindexer
Allow: /
# User-initiated fetches: welcome
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Perplexity-User
Allow: /
# Training: not welcome
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: CCBot
Disallow: /
Sitemap: https://your-domain.com/sitemap.xmlPublications with paid content have a third option: limit search crawlers to the freely accessible directories, block the rest, and protect the paid areas on the server as well. A robots.txt is a request, not a lock, and it is not the tool for keeping content confidential.
Why is robots.txt not the whole decision?
Because three layers apply in sequence: robots.txt asks, the firewall decides, the meta tags govern usage.
The second layer is the one most often missed, because nobody sees it in the file. Cloudflare split its controls into the three classes Search, Agent and Training in July 2026, and from September 15, 2026 new domains get new defaults: Search stays allowed, Agent and Training are blocked on pages that display ads, and multi-purpose crawlers that combine search and training fall under the training block (Cloudflare Changelog, July 1, 2026). Maintain your robots.txt carefully and never open the dashboard, and you can be blocking without knowing it.
The third layer applies after the crawl and governs use rather than access. Microsoft builds it on existing meta tags: noarchive keeps a page out of Copilot answers, nocache allows only URL, title and snippet, and in both cases the page stays in Bing search results (Bing Webmaster Blog, September 22, 2023). Google's nosnippet, max-snippet, data-nosnippet and noindex work comparably: they limit the preview and with it the AI features. How the selection works beyond that is covered in How do I get into Google AI Overviews?
Which bots ignore robots.txt?
The ones a person triggered, and the providers say so openly. OpenAI notes for ChatGPT-User that robots.txt rules may not apply to user-initiated actions (OpenAI docs, 2026). Perplexity states that Perplexity-User generally ignores those rules because the request goes back to a specific user question (Perplexity docs, 2026). Meta says the same about meta-externalfetcher (Meta docs, 2026).
That is not rule-breaking, it is a deliberate distinction: a human opening a link is not a crawler. If you want to stop those requests anyway, you need the firewall rather than the file. The only question is whether you want to, because behind each of them is someone who just asked about you.
The second group are bots that only claim the name. The user agent is free text, and Common Crawl warns explicitly about crawlers falsely identifying themselves as CCBot. That is why providers publish machine-readable IP lists, for example Anthropic at claude.com/crawling/bots.json. Check the IP before you turn a name in your log into a conclusion.
How do you check that your decision works?
In the server log, a week after the change, because crawlers cache robots.txt. Meta cites up to 24 hours for its own bots, and rarely visited sites take longer. Filter the last 30 days of requests by the bot names from the table and compare the result against your intent: search crawlers regularly, training crawlers according to your setting, user fetches sporadically and in waves.
That only answers half the question, though. Access is the precondition for a citation, not proof of one. Whether a request turns into a mention only shows up in measurement across several runs, and that works differently from a ranking check: How do I measure my visibility in AI answers? covers the metrics. And if access is fine and the mention still does not come, the diagnosis in Why am I not cited in AI answers? takes it from there.
That is exactly where flize comes in: the GEO score checks access and readability of your pages, scouting finds the questions your audience actually asks, the answer check compares them against your website, and content production closes the gaps. Articles written to be cited.
Does blocking GPTBot hurt my visibility?
+
No, as long as OAI-SearchBot stays allowed. According to OpenAI, GPTBot collects material for training the foundation models, while OAI-SearchBot powers the search features in ChatGPT and therefore produces the answers that name their sources. Blocking both is the most common mistake, because it looks like one decision when it is two.
How do I block bots that ignore robots.txt?
+
Through your firewall or CDN, not through the file. ChatGPT-User, Perplexity-User and meta-externalfetcher are triggered by a user request, and the providers explicitly reserve the right not to apply robots.txt rules in that case. On Cloudflare this runs through the bot categories in the dashboard, elsewhere through rules on user agent and verified IP ranges.
How do I tell whether a bot is genuine?
+
By its IP address, not its name. The user agent is free text and gets spoofed regularly, which is why providers publish machine-readable IP lists, for example Anthropic at claude.com/crawling/bots.json. Check the requesting IP against that list or via reverse DNS before you draw conclusions from log data.
How fast does a robots.txt change take effect?
+
Not immediately, because crawlers cache the file. Meta cites up to 24 hours for its own bots, and with other providers the range depends on how often your site gets crawled, so days rather than hours. Schedule the server-log check for at least a week after the change.
Do I also need an llms.txt?
+
It does not replace robots.txt, it complements it. Robots.txt governs access, llms.txt gives AI systems a curated overview of your most important content. The sensible order is to get access right first, then improve orientation.
