Search and AI
AI crawlers and robots.txt: what to allow, what to block, and how it is read
Blocking AI crawlers is one line per crawler in robots.txt. The care is in choosing which: the same company runs a crawler for its search and a separate one for training, and blocking the wrong one takes you out of answers you wanted to be in.
By Oscar CobbeCurrent as at 3 minute read8 sources
Which crawler does what
Each company's own documentation, in short. The search crawlers decide whether you can be found and cited; the training crawlers decide whether your pages may be used to train models; the user fetchers open a page because somebody asked about it.
| Company | Search | Training | Opened when a person asks |
|---|---|---|---|
| Googlebot (Search, AI Overviews, AI Mode) | Google-Extended (Gemini; Google says it does not affect Search) | None separate | |
| OpenAI | OAI-SearchBot (ChatGPT search) | GPTBot | ChatGPT-User (robots.txt may not apply) |
| Anthropic | Claude-SearchBot | ClaudeBot | Claude-User |
| Perplexity | PerplexityBot (Perplexity says it is not used for training) | None named | Perplexity-User (generally ignores robots.txt) |
| Apple | Applebot (Siri, Spotlight, Safari) | Applebot-Extended (fetches nothing itself) | None separate |
| Microsoft | bingbot | None named | None separate |
| Meta | meta-webindexer (Meta AI search) | meta-externalagent | meta-externalfetcher (may bypass robots.txt) |
How a crawler reads the file
RFC 9309, the robots exclusion standard, sets three rules that catch people out. A crawler obeys the group of rules that names it and ignores the * group altogether, so naming a crawler to block it means repeating anything else you wanted it to follow. Two groups naming the same crawler are merged. And within a group the longest matching rule wins, with Allow beating Disallow at the same length.
A missing robots.txt means everything may be read. A robots.txt the server cannot serve, because it errors, means a crawler that follows the standard reads nothing until it can.
A file that keeps search open and opts out of training
This is the shape to copy if you want to be found and cited but not used for training. Each block names one crawler; a crawler not named falls through to the last block.
- 1User-agent: GPTBot, then Disallow: /
- 2User-agent: ClaudeBot, then Disallow: /
- 3User-agent: Google-Extended, then Disallow: /
- 4User-agent: Applebot-Extended, then Disallow: /
- 5User-agent: meta-externalagent, then Disallow: /
- 6User-agent: CCBot, then Disallow: /
- 7User-agent: *, then Allow: /, then the line Sitemap: followed by your sitemap's full address
OpenAI says each setting is independent
OpenAI's crawler page gives this exact case as an example: a site can allow OAI-SearchBot, and appear in ChatGPT search, while disallowing GPTBot, which signals its pages should not be used to train OpenAI's models.
What robots.txt cannot do
It asks; it does not enforce. The companies above say their crawlers follow it, but a request a person triggered may not, by their own account, and a crawler that does not identify itself is not covered by any of this.
It controls crawling, not indexing. Google says a page blocked in robots.txt can still appear in its results without a description if other sites link to it, and that a noindex rule on a blocked page is never seen. To keep a page out of Google, allow it to be crawled and mark it noindex.
And a hosting firewall or a security service can block crawlers before they ever ask for the file, which robots.txt cannot show you. If a search crawler is allowed and still never reaches your pages, that is the next place to look.
Check what yours says
The free AI crawler access check reads your robots.txt the way the standard says and lists each crawler above as allowed or blocked for your homepage, with the rule that decided it. If you would rather we looked with you, book a 20-minute call.
Sources
- 1.RFC 9309, Robots Exclusion Protocol · Internet Engineering Task Force
- 2.Overview of OpenAI crawlers · OpenAI
- 3.Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic
- 4.Perplexity crawlers · Perplexity
- 5.Google's common crawlers · Google Search Central
- 6.Block search indexing with noindex · Google Search Central
- 7.About Applebot · Apple
- 8.Meta web crawlers · Meta
Free, and no account needed
See which crawlers your robots.txt lets in
Fetches your robots.txt and lists Googlebot, bingbot, OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot, Applebot and the rest as allowed or blocked, with the rule that decided each.
Run the free checkNot sure which to block?
Book a 20-minute call and we will read your robots.txt with you and say what to change first.
Book a 20-minute callWho wrote this
Oscar Cobbe · Founder, FourWinds Digital
Writes and maintains the legal explainers on this site, and does the compliance work behind them. Every date and article number here is checked against the legislation itself before it is published, and each article shows the date it was last checked.
More about how we work →Read next
- Search and AIHow Google's AI features, ChatGPT and other assistants find and cite a business
- Search and AIllms.txt: what it is, who says they read it, and whether to bother
- Search and AIWhy is my website not showing on Google? Six checks, in order
Written on 29 October 2026; it describes the law as FourWinds read it on that date. This is general information about how the rules work, not legal advice on your situation. Where you need a solicitor, we say so. How we check the law.