llms.txt — Does It Actually Do Anything Yet?

As of today, no major AI company has publicly confirmed that its systems read your site's llms.txt file. Google's own search documentation says the opposite of what most llms.txt advice implies: "You don't need to create new machine readable files, AI text files, or markup to appear in these features."

That is the honest state of it, and it is worth saying plainly before the detail, because the file is being sold as a ranking factor for AI search and there is no public evidence that it is one.

None of which makes it useless. It does something real in one specific context, it costs almost nothing if you generate it properly, and we publish one at /llms.txt ourselves. This post is about keeping those three claims separate — what the proposal says, what is confirmed, and what actually moves AI citations.

What does the proposal actually specify?

llmstxt.org is a proposal by Jeremy Howard, first published in September 2024 and revised to v2 in August 2026. Its reasoning is about context windows rather than ranking: an HTML page wraps its information in navigation, ads and JavaScript, converting that back to clean text is "difficult and imprecise", and "every wasted token costs time and money".

The v2 spec is short and precise. A file at /llms.txt — or at any subpath, covering the pages beneath it — containing, in order:

• An H1 with the name of the project or site. This is the only required section.
• A blockquote with a short summary.
• Any number of non-heading markdown sections giving more detail.
• Any number of H2 sections containing "file lists": markdown links, each optionally followed by a colon and a note.

By convention a section headed ## Optional holds links an agent can skip when it needs a shorter context.

There is a second half that gets much less attention and is arguably the more useful one. The proposal also asks sites to serve a clean markdown version of each page at the same URL with .md appended or substituted (page.md, index.md), and to advertise both with standard link relations — rel="alternate" type="text/markdown" pointing at the markdown version, rel="describedby" pointing at the llms.txt that covers it. Those can be HTML <link> elements or an HTTP Link: header, which means a CDN can add them without touching a single page.

The spec is also clear that llms.txt is not a replacement for anything. It expects to coexist with robots.txt, which is about access, and with sitemap.xml, which lists every indexable page rather than offering a curated overview. It explicitly addresses why it did not use /.well-known/ (RFC 8615): well-known URIs exist only at the origin root, and plenty of authors control only a path on a shared host.

It is a sensible, modest proposal. It has simply been retold as something it never claimed to be.

Who has confirmed reading it?

This is the question that decides whether to spend time on it, so here is what each party has actually said.

Google. The clearest position of anyone. Its AI features and your website documentation states that you do not need to create new machine-readable files or AI text files to appear in AI Overviews and AI Mode, and that there is no special structured data for them either. Separately, in June 2025, Google's John Mueller said no AI system currently uses llms.txt and compared it to the keywords meta tag — a file site owners fill in that nothing consumes.

OpenAI. Its crawler documentation tells you to manage OAI-SearchBot and GPTBot through robots.txt. It does not mention reading your llms.txt. OpenAI does publish an llms.txt for its own developer docs, which is a different claim entirely.

Anthropic. Its crawler help page documents ClaudeBot, Claude-User and Claude-SearchBot and says the bots honour robots.txt directives. llms.txt is not mentioned anywhere on it.

Perplexity. Its crawler documentation documents PerplexityBot and Perplexity-User, again via robots.txt, and recommends allowing PerplexityBot so your site can appear in results. No mention of reading your llms.txt — though, like the others, its own docs site publishes one.

So: four providers, four crawler documents, four pointers to robots.txt, and no stated consumption of llms.txt. Three of the four publish one for their own documentation, which tells you they think the format is useful for coding agents reading docs. It does not tell you their retrieval systems read yours.

Anyone claiming otherwise should be asked for the first-party link. There currently isn't one.

Where it demonstrably does work

One place, and it is genuinely worth knowing about: coding agents reading documentation.

This is the use case the proposal was built for and the one where adoption is real. Documentation platforms generate llms.txt automatically, the AI labs publish files for their own developer docs, and an agent that needs an exact API signature benefits concretely from a curated index of markdown pages over scraping a JavaScript-rendered docs site. If you ship a developer-facing product or an SDK, this is a legitimate reason to publish one, and it has nothing to do with marketing.

There is also a second, weaker reason that has appeared recently. Chrome's Lighthouse now has an Agentic Browsing audit category that, among other checks, "checks for the presence of a machine-readable summary at the domain root". Read Google's own framing before you treat that as a mandate: the category is "experimental and based on proposed standards", requires Chrome 150 or later, and deliberately has no 0–100 score, because "the standards for the agentic web are still emerging". It also sits alongside WebMCP and accessibility-tree checks, which are about agents operating your site rather than citing it. It is informational. It is not an SEO signal, and Google Search's documentation — quoted above — says the opposite in the same breath.

llms.txt against the files that are actually read

File	What it does	Who is documented as reading it
robots.txt	Controls crawler access per user agent	Googlebot, OAI-SearchBot, GPTBot, PerplexityBot, ClaudeBot, Claude-SearchBot — all documented by their owners
sitemap.xml	Lists indexable URLs for discovery	Search engines, documented
Schema.org structured data	Describes entities on a page	Search engines for rich results; Google says none is required for AI features
llms.txt	Curated markdown overview for agents	No major AI provider has publicly confirmed consuming it; coding agents and docs tooling do

The first row is where the leverage is, and it is the row people skip because it is dull.

What actually affects whether AI assistants cite you

Three things, in descending order of how often we find them broken.

1. Crawler access. Every provider gates its search surface behind a specific user agent, and these are routinely blocked by accident — a security plugin's bot-blocking preset, a Cloudflare managed rule, a staging-era Disallow: / that survived launch. OpenAI documents that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers". Perplexity recommends allowing PerplexityBot and permitting its published IP ranges. Anthropic says disabling Claude-SearchBot prevents indexing for search. Note also that each provider separates its search bot from its training bot, so you can allow one and refuse the other, and that both OpenAI and Perplexity document a roughly 24-hour lag before a robots.txt change takes effect.

Check this before anything else. A site invisible to OAI-SearchBot cannot be fixed by a markdown file.

2. Content a retrieval system can use. Google's AI features documentation lists what still matters, and it is unglamorous: crawling allowed in robots.txt "and by any CDN or hosting infrastructure", internal links that make content findable, good page experience, "important content available in textual form", and structured data that matches the visible text. Nothing in that list is AI-specific. The practical version: if your pricing, your answers and your specifics exist only inside an image, a PDF carousel or a client-rendered widget, no amount of agent-friendly metadata rescues them.

3. What third parties say about you. Assistants ground answers in what their retrieval layer finds, and that is frequently not your site. Directory listings, comparison pages, review sites, forum threads and trade publications all get read. For Indian service businesses this is usually the largest practical gap, and it is slow, unexciting work: an accurate Business Profile, listings that agree with each other, and genuine third-party coverage. There is no file you can publish that substitutes for being described consistently elsewhere on the web.

This is the substance of GEO work, and we price it openly precisely because a lot of what is sold under that label is a file upload and a dashboard.

So should you publish one?

A reasonable rule:

Yes, if you ship developer documentation, an SDK or an API, or you already generate your site's content from structured data so the file costs you nothing to keep current. Both are real cases.

Probably not worth prioritising if it would be hand-maintained. A stale llms.txt is worse than none: it quietly misrepresents your site to anything that does eventually read it, and it will go stale, because every hand-maintained index does.

We know this one first-hand. Ours used to be a hand-written file sitting in public/, and it drifted — it shipped prices that disagreed with every pricing page, and its page index listed four of eight comparison guides and none of the location pages added after it was written. It now generates from the same data the sitemap reads, which means the index and the prices cannot fall out of step with the site again. That was the fix worth making. The file's existence was never the point.

Definitely not as a replacement for any of the three things in the previous section, and not because someone told you it improves AI rankings.

What not to do

Do not pay for llms.txt implementation as a line item in an AI search package. Generating one from content you already have is a short piece of work; if a proposal prices it as a deliverable, ask what evidence the vendor has that anything reads it.

Do not block crawlers and publish an llms.txt. We see this combination surprisingly often — a security configuration refusing OAI-SearchBot while the site advertises a lovingly curated markdown index. It is a locked door with a menu taped to it.

Do not generate llms-full.txt files running to hundreds of thousands of tokens and expect them to be read. The proposal's whole argument is that the file "stays small enough to fit in context", with detail behind links fetched only when needed. A dump of your entire site inverts that.

And do not treat the current answer as permanent. This is a two-year-old proposal with a v2 that landed last month, Chrome shipping experimental audits around it, and no confirmed consumption by the major providers. Any of that could change in a quarter. The reason to be sceptical now is the absence of evidence, not a belief that the idea is bad — which is also why it is worth re-checking rather than deciding once.

If you want the actual answer for your site, the useful version is a short audit: what your robots.txt permits per provider, whether your substance exists as text, and what third parties currently say about you. Tell us what you have and we will tell you which of those three is costing you citations — and if the answer is that a missing llms.txt is the least of your problems, we will say that instead of selling you one.