Which search and AI crawlers can read your website?
Enter your website and see which of Google's, Bing's, ChatGPT's, Claude's, Perplexity's and Apple's crawlers your robots.txt lets read your homepage, and what each one is for.
Free and no account. The domain you type goes to our server, which runs the check and counts it against an hourly limit.
How it works
The check fetches three addresses on your domain: /robots.txt, /llms.txt and the homepage. It reads robots.txt the way RFC 9309, the robots exclusion standard, says a crawler should. A crawler obeys the group of rules that names it and ignores the rest; one that is not named obeys the rules for all crawlers (*); the most specific matching rule wins. With no robots.txt at all, every crawler may read everything. With one the server cannot serve, a crawler that follows the standard reads nothing.
Each crawler is described from its owner's documentation, linked beside it. Some feed a search engine, some open a page when a person asks an assistant a question, and some collect pages to train AI models. Blocking a training crawler does not take you out of that company's search: OpenAI, Google and Apple each say so of their own.
Google says a page needs nothing beyond normal indexing to appear in AI Overviews and AI Mode: no special file and no special markup. A page has to be indexed and able to show a snippet. How assistants find and cite a business goes further into what decides that.
What it cannot tell you
- It checks your homepage's address. Rules for other folders can be different.
- A firewall or a hosting service can turn a crawler away before it ever asks for robots.txt. This check reads the file and cannot see that.
- Allowing a crawler promises nothing. No search engine or assistant guarantees to show or cite a page, and nothing here claims one does.
Sources
- 1.RFC 9309, Robots Exclusion Protocol · Internet Engineering Task Force
- 2.AI features and your website · Google Search Central
- 3.Google's common crawlers · Google Search Central
- 4.Overview of OpenAI crawlers · OpenAI
- 5.Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic
- 6.Perplexity crawlers · Perplexity
- 7.About Applebot · Apple
- 8.Meta web crawlers · Meta
- 9.CCBot · Common Crawl
- 10.Which crawlers does Bing use? · Microsoft Bing
Read next
- Search and AI
How Google's AI features, ChatGPT and other assistants find and cite a business
- Search and AI
llms.txt: what it is, who says they read it, and whether to bother
- Search and AI
AI crawlers and robots.txt: what to allow, what to block, and how it is read
- Search and AI
Why is my website not showing on Google? Six checks, in order
Not sure what the result means for you?
Book a 20-minute call and we will go through it with you, with the site in front of us.
Book a 20-minute call