llms.txt generator

It's a plain Markdown file at the root of your site — /llms.txt — that gives a language model a short, hand-picked map of your most important pages. It starts with your site name as an H1, a one-line summary, then grouped lists of links, each as a Markdown hyperlink with an optional note. The idea, proposed by Jeremy Howard of Answer.AI in 2024, is that a model handed this file can find your key content without crawling and guessing. Think of it as a curated table of contents written for machines, not an exhaustive URL dump.

No, and we won't pretend otherwise. It's a proposed convention from one author, not a ratified standard like a W3C or IETF spec, and no search engine or AI company has formally adopted it. Some well-known sites publish one (Anthropic, Cloudflare, Stripe, and every docs site hosted on Mintlify, which generates the file automatically), but publishing a file is not the same as the major models reading it. Treat it as a low-cost, forward-looking convention, not a guaranteed win.

There's no evidence it does, and Google has said it doesn't use the file. Google's John Mueller compared it to the old keywords meta tag and noted that server logs show the AI crawlers don't even request it. So don't add an llms.txt expecting a ranking bump. That's not what it's for. Add it because it's a cheap, tidy way to describe your best content for any tool that does choose to read it.

No major AI vendor has confirmed it reads llms.txt during normal retrieval, and published server-log analyses show their crawlers don't routinely request the file. The honest framing: an llms.txt makes your content easier for a model to use if the file is given to it — for example, pasted into a prompt or fed by a tool that supports the convention — not a switch that makes every AI read your site differently. Adoption may grow; today it's early.

Save the generated text as a file named llms.txt and serve it at your site root, so it loads at https://yourdomain.com/llms.txt. Most static hosts let you drop it in the public folder; on a CMS you may need a redirect or a plugin. The spec allows a subpath too, but root is the convention everyone checks first.

No, and conflating them is a common mistake. robots.txt controls which crawlers may access which paths, and crawlers honor it. llms.txt does the opposite job: it's a content guide, not an access rule. Nothing is blocked or enforced by an llms.txt, and it doesn't replace robots.txt. If you want to allow or block AI crawlers, that's a robots.txt question, not this file.

llms.txt is the concise index this tool builds — names, links, short notes. llms-full.txt is a separate, larger file that inlines the full text of your pages so a model can ingest everything in one fetch. Worth knowing: llms-full.txt is a popular community convention (Mintlify and others generate it) but it is not part of the llmstxt.org spec itself, which only defines llms.txt. This generator produces the index file.

Treat it as a strong first draft, not a final file. We detect your site name and summary from your homepage, pull pages from your sitemap (or your homepage links if there's no sitemap), group them into sections, and fetch real page titles for the first batch. You should still read it, drop pages that don't belong, tighten the summary, and move low-value pages under the Optional heading. The point of llms.txt is that a human curated it.