How Google's AI features, ChatGPT and other assistants find and cite a business

Every AI assistant that cites websites reads them with a crawler, and every company that runs one publishes rules for it. None of them promises to cite anybody. What their own pages say a business can do is short, and most of it is ordinary good practice.

By Oscar CobbeCurrent as at 29 October 20264 minute read8 sources

Three kinds of crawler

The companies' own pages describe three different jobs, often done by three different crawlers from the same company.

1Search crawlers read pages ahead of time and keep an index the assistant answers from: Googlebot for Google Search and its AI features, OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity, Applebot for Siri and Spotlight.
2Fetchers open one page at the moment a person asks a question that needs it: ChatGPT-User, Claude-User, Perplexity-User. OpenAI and Perplexity both say robots.txt may not apply to these, because a person asked.
3Training crawlers collect pages that may be used to train future models: GPTBot, ClaudeBot, and Google-Extended and Applebot-Extended, which decide how pages already crawled may be used. Blocking these does not take you out of the same company's search: OpenAI, Google and Apple each say so.

Google: no special file, no special markup

Google's page on AI features and your website says a page must be indexed and eligible to be shown in Google Search with a snippet to appear as a supporting link in AI Overviews or AI Mode, and that there are no additional requirements and no special optimisations needed for either. It says plainly that you do not need new machine-readable files, AI text files or special schema.org markup.

The controls it names are the ordinary ones: nosnippet, data-nosnippet and max-snippet to limit what is shown, and noindex to keep a page out. AI feature traffic is counted inside the normal Search Console Performance report, under the Web search type, and is not broken out separately.

ChatGPT, Claude and Perplexity

OpenAI's crawler page says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Anthropic says blocking Claude-SearchBot may reduce a site's visibility in Claude's search results. Perplexity says PerplexityBot surfaces and links websites in its search results and is not used to train foundation models.

In each case the rule that decides access is your site's robots.txt. The free AI crawler access check reads yours and lists each crawler as allowed or blocked, with its owner's description beside it; the robots.txt guide shows how to let the search crawlers in while keeping the training crawlers out.

What a business controls

Nothing on this list is specific to AI, which is the point Google's own page makes.

1Access: the search crawlers above are allowed in robots.txt, and nothing in front of the site, such as a firewall rule, turns them away.
2Indexing: the pages that matter carry no noindex, are in your sitemap, and are linked from other pages on the site. Why a site is not on Google covers the checks.
3Plain facts on the page: what you do, where you do it, for whom and, where you can, for what price, in sentences a reader could quote.
4Consistent listings: the name, address or service area and website on your Google Business Profile match the site. Setting up a profile with no shopfront follows Google's own rules.

llms.txt

llms.txt is a proposed file, published since September 2024, that gives AI agents a short Markdown map of a site. Its own page claims no commitment from any search engine or AI company to read it, and Google says no AI text files are needed for its AI features. We publish one, because it costs little and an agent may use it; what it is, and what it is not.

Measuring it, honestly

Search Console counts Google's AI features within your ordinary search figures, so there is no separate line for them. Google Analytics 4 has a default AI Assistant channel for visits referred by an assistant. On this site it counted 2 sessions in the 90 days to 28 October 2026, against 52 from organic search. That is our figure for our site, and it says nothing about anybody else's.

What no tool can show is how often an assistant mentioned you without a click, or what it said. Any service that reports that is reporting its own sample of answers, not the assistant's.

Sources

Free, and no account needed

Which crawlers can read your site?

Reads your robots.txt and tells you which search and AI crawlers may fetch your homepage, whether it asks not to be indexed, and whether you publish a sitemap and an llms.txt.

Run the free check

A website that says what you do, plainly

A landing page or a full site that loads fast on a phone and gets every enquiry to you, with your Google Business Profile set up to match it, and the code and the domain handed over in your name.

See the Websites page

Who wrote this

Oscar Cobbe · Founder, FourWinds Digital

Writes and maintains the legal explainers on this site, and does the compliance work behind them. Every date and article number here is checked against the legislation itself before it is published, and each article shows the date it was last checked.

More about how we work →