What is llms.txt — and does your site actually need one?

Someone drops a message in your team chat: "We need an llms.txt. Every AI tool is about to start reading it, and we're behind." There's a vague panic to it, the same one that drove people to stuff keywords into meta tags in 2009. So you go looking for a straight answer to a simple question — do I actually need this thing? — and you find a wall of breathless posts telling you it's the future, and not one of them showing proof it does anything.

Let me save you the afternoon. We built a free llms.txt generator, so this is a little awkward to admit: when I went hunting for evidence the file gets read, I came up nearly empty — and the data, it turns out, is even less flattering than that. Here's what llms.txt really is, what the spec actually defines (and what it doesn't), who's using it, and the honest decision rule for whether you should bother. Date-stamped on purpose — this is the state of things as of June 2026, and it's a topic that moves.

What is an llms.txt file?

llms.txt is a plain Markdown file you put at the root of your site — yoursite.com/llms.txt — that hands a large language model a curated map of your most important content. It was proposed by Jeremy Howard of Answer.AI in September 2024, and the spec's stated purpose is to "provide information to help LLMs use a website at inference time" — that is, when a model is answering a question right now, not when it's being trained.

The format is deliberately tiny. One required H1 with your project name. An optional one-line summary as a blockquote. Some optional free text. Then zero or more H2 sections, each a bullet list of links shaped like - [Page name](https://url): short note. That's the whole thing. The idea is that instead of making a model crawl and guess at your sprawling site, you hand it a clean index of what matters: here are the docs, here's the pricing, skip the rest.

It's a nice idea. Tidy, human-readable, easy to generate. The trouble isn't the idea. It's whether anything on the other end is actually reading it.

Does llms.txt actually work yet?

This is the question the hype posts skip, so here's the blunt answer: as of mid-2026, no major AI provider — OpenAI, Anthropic, Google, Perplexity, Microsoft — has confirmed that its crawlers or models read other sites' llms.txt files. And the largest study we have suggests they overwhelmingly don't.

Ahrefs analyzed 137,210 domains in May 2026 (published 15 June 2026). About 28% of them had published an llms.txt. And 97% of those files received zero requests that month — not from AI bots, not from anyone. The same sites were getting tens of millions of AI-crawler hits overall; the bots were crawling the sites heavily and ignoring the llms.txt sitting right there at the root. The file isn't being declined. It's being walked straight past.

Google has been the most direct about it. Back in 2025, John Mueller put it this way on a TechSEO thread:

"AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag."

That comparison stings on purpose. Gary Illyes, also at Google, has said Google doesn't support llms.txt and has no plans to, and that ordinary SEO is what surfaces you in AI Overviews (Search Engine Land, 2025). Mueller came back to it again in June 2026, noting that a model "by design can't trust" a file a site writes about itself — which is the deeper problem. Anyone can claim anything in their own llms.txt. Why would a model lean on that over what it can verify by reading the actual pages?

llms.txt vs robots.txt vs sitemap.xml: which ones are real?

People mix these three up constantly, and the distinction is the fastest way to understand why llms.txt is in a different category. The cleanest way to hold it: robots.txt gates, sitemap.xml enumerates, llms.txt curates — and only the first two are actually honored.

robots.txt is an access-control file. It tells crawlers which paths they may and may not touch, and the well-behaved ones obey it. (Whether to use it to block AI crawlers is a real decision with real tradeoffs — we get into that in should you block GPTBot, ClaudeBot and Google-Extended.) sitemap.xml is a discovery file: a complete inventory of your URLs so search engines can find and index everything. Both are supported standards that the big crawlers genuinely use.

llms.txt does a third, different job — it curates a short list of your best content as a reading suggestion. Nothing is blocked, nothing is enumerated exhaustively, nothing is enforced. And unlike the other two, no major engine has committed to reading it. So if you're choosing where to spend an hour, a correct robots.txt and a current sitemap.xml beat an llms.txt every time. Those get used today. (Our sitemap analyzer is the boring-but-useful place to start on the one that actually counts.)

Then why do Stripe, Anthropic and Cloudflare all have one?

Fair question, and it's the strongest argument for bothering. A lot of serious companies publish llms.txt — Anthropic, Stripe, Cloudflare, Vercel, Supabase, Cursor. So am I wrong?

Two things explain most of it. First, a big share of those are documentation sites hosted on Mintlify, which has auto-generated an llms.txt for every docs site it hosts since around November 2024. The companies didn't choose it; the file showed up for free. Second — and this is the honest case for it — dev-tool docs are exactly the use case where it might genuinely pay off someday. When developers ask an AI assistant "how do I authenticate with this API," a clean, curated docs index is precisely the thing that would help, if the tools start reading it. So the adopters skew heavily toward API docs, not random marketing sites, and that's not an accident.

One small correction while we're here, because it trips people up: the file you'll sometimes see called llms-full.txt (one big concatenated dump of all your docs) is not part of Howard's spec. It's a popular companion convention that Mintlify and a few adopters made up. Useful, fine to have, just don't go hunting for it in the official spec; it isn't there.

So should you add one? An honest decision rule

Here's how I'd actually decide, and I'll keep it simple because the file doesn't deserve more agonizing than this.

Add one if you run developer documentation, an API, or a large reference site, and generating it is nearly free (your platform does it, or our generator does it in a few seconds). The downside is essentially zero, and you're cheaply positioned for the day — if it comes — when the tools start honoring it. Think of it as a tidy business card you leave at the door, not a billboard.

Skip it, or at least don't prioritize it, if you're a small marketing site or blog hoping for an AI-traffic bump. There's no evidence it delivers one, Google has said outright it won't help your rankings, and the file mostly isn't being fetched. If your goal is to show up in AI answers, your time goes to the things that actually move that — clear, quotable content and brand mentions, which I cover in how to get cited by ChatGPT, Perplexity and AI Overviews — not into a file nobody's reading.

The one thing I'd genuinely warn against: don't add an llms.txt and then tell your boss the AI-visibility problem is handled. It isn't. That's the failure mode the keywords meta tag created — a checkbox that felt like progress and did nothing.

How do you create and publish one?

If you've decided it's worth the five minutes: you can write it by hand in any text editor (it's just Markdown — H1 with your name, a summary line, then H2 sections listing your key pages), or generate a starter automatically. Either way it goes at your site root, reachable at yoursite.com/llms.txt, served as plain text.

Our llms.txt generator reads your homepage and sitemap and builds a spec-compliant draft you can edit down — which saves the tedium, though the judgment of which pages are truly your best is still yours to make; a tool can't know that. And to be clear about what I'm sending you to: that generator builds a clean file, not a ranking boost. We built it because it's a reasonable thing to have, not because it's a growth lever. Same honesty I'd want from anyone pointing me at their own tool.

Questions people ask

Will an llms.txt get me cited by ChatGPT or show up in AI Overviews?

There's no evidence it will, and Google has explicitly said the file won't help you appear in AI Overviews. No major AI provider has confirmed reading other sites' llms.txt, and a 137,000-domain study found 97% of published files got zero requests in a month. If being cited by AI is the goal, structure and brand mentions are what move it — the file isn't a citation lever. Don't add it expecting AI visibility.

Does llms.txt affect my Google rankings?

No. Google has said directly that it doesn't use llms.txt and there are no plans to, and that normal SEO is what surfaces content in AI Overviews. It won't help your rankings and it won't hurt them. Treat it as a content-curation convention for AI tools that might one day read it, not as anything search-related.

Is llms.txt the same as a sitemap?

No. A sitemap.xml lists every URL on your site so search engines can discover and index all of it, and search engines genuinely use it. An llms.txt curates a short, hand-picked list of your most important pages as a reading hint for language models, and no major model has committed to reading it. A sitemap is an exhaustive inventory that's actually honored; llms.txt is a selective suggestion that mostly isn't.

Where does the llms.txt file go?

At the root of your domain, reachable at yoursite.com/llms.txt, served as plain text. Same location pattern as robots.txt. If you publish the bigger concatenated version, that convention lives at yoursite.com/llms-full.txt — but note that's an adopter convention, not part of the official spec.

Do I need llms-full.txt as well?

Only if you run large documentation and want to offer a single concatenated copy of it for tools that prefer one big file. For most sites it's overkill, and it isn't part of the original spec — it's a convention popularized by Mintlify and a few others. If you're a blog or marketing site, you almost certainly don't need either file, let alone both.

The honest takeaway

llms.txt is a thoughtful proposal that costs nearly nothing to adopt and, as of mid-2026, demonstrably almost nobody's AI is reading. That's not a reason to mock it — conventions take time, and the dev-docs use case is real. It's a reason to keep it in proportion: a cheap, tidy file to have if you run documentation, and a non-event for everyone hoping it's a shortcut to AI visibility. There is no shortcut. There's just clear content, in places the engines already trust.

If you do want one without learning the spec yourself, generate a starter llms.txt from your homepage and sitemap in a few seconds, then edit it down. And if you're weighing the bigger AI-search questions behind it, the two that actually matter more are whether to let the AI crawlers in at all and what genuinely earns you a citation — both of which do more for your visibility than this file ever will. And if you've seen Google's new Open Knowledge Format mentioned in the same breath as llms.txt, that's the inward-facing cousin of this idea — same hype, and the same gap between the claims and what actually reads it.