llms.txt: what it is, who says they read it, and whether to bother

llms.txt is a short Markdown file at the root of a website that points AI agents at the pages worth reading. It is a proposal, not a standard, and no search engine or AI company has said it uses one to decide what to cite.

By Oscar CobbeCurrent as at 29 October 20263 minute read3 sources

What it is

Jeremy Howard proposed llms.txt on 3 September 2024, and the proposal is maintained at llmstxt.org. The idea is that an AI agent reading a website wastes effort on menus, scripts and layout, so the site offers it a plain summary with links to the pages that matter, ideally in a clean text or Markdown version.

The file lives at /llms.txt, though the proposal allows one at any path for the pages under it. It is written in Markdown, in a fixed order:

1A heading with the name of the site or project. This is the only part that is required.
2A short summary, as a block quote.
3Optional paragraphs or lists with more detail, with no headings.
4Sections, each with its own heading, listing links with a note on each.
5By convention, a section called Optional, for links an agent can skip when it needs less.

Who reads it

The proposal describes what agents are expected to do with the file. It does not say that any search engine or AI company has committed to reading it, and none of the crawler pages we read for how assistants find a business mentions it.

Google is direct about its own AI features: its page for site owners says you do not need new machine-readable files or AI text files to appear in AI Overviews or AI Mode. A page has to be indexed and able to show a snippet, and that is all.

Where llms.txt is used is closer to the proposal's intent: developer documentation, read by coding assistants that are pointed at it. The proposal notes that OpenAI, Anthropic and Google publish llms.txt files for their own developer docs.

Is it worth doing?

For a small business site it is a short job, and it does no harm. It is also no substitute for what decides whether any assistant can use your pages: crawlers allowed in robots.txt, pages indexed, and plain facts written on the pages themselves.

We publish one on this site. It is built from the same catalogue as the product pages, so the prices in it cannot drift from the prices on the site, and it lists the guides and tools an assistant is most likely to be asked about. You can read it. Nothing we measure shows whether it has led to a visit or an enquiry, and nothing here says it will for you.

Do the access check first

An llms.txt on a site whose robots.txt blocks the search crawlers is a map nobody is allowed to follow. The free AI crawler access check reads both.

If you write one

Keep it to what a stranger would need: who you are, what you do and where, your prices if you publish them, and links to the pages that answer the common questions. Write the notes as facts. Keep it in step with the site, because an assistant that reads an out-of-date price in it will repeat the out-of-date price.

Check it is served as plain text at yourdomain/llms.txt. The AI crawler access check reports whether it finds one.

Sources

Free, and no account needed

Does your site publish one, and can crawlers reach it?

Reads your robots.txt, your homepage and /llms.txt, and tells you which search and AI crawlers may fetch your site and whether the file is there.

Run the free check

Want a second opinion on your site?

Book a 20-minute call and we will go through what search engines and assistants can read on your site, and what to change first.

Book a 20-minute call

Who wrote this

Oscar Cobbe · Founder, FourWinds Digital

Writes and maintains the legal explainers on this site, and does the compliance work behind them. Every date and article number here is checked against the legislation itself before it is published, and each article shows the date it was last checked.

More about how we work →