What is llms.txt and who needs it?
llms.txt is a Markdown file in the root directory of a website (your-domain.com/llms.txt) that gives AI systems a curated overview of the most important content: what the site is, which pages matter and where they live. The proposal comes from Jeremy Howard (Answer.AI, September 2024). Unlike robots.txt it does not control access; it helps language models understand.
- A Markdown file at /llms.txt: short description plus a curated link list.
- robots.txt controls access, sitemap.xml lists everything, llms.txt curates what matters.
- Effort: under an hour. Risk: practically none.
- Not an official standard (yet), but a growing ecosystem.
What goes into an llms.txt?
The format is deliberately simple and made for language models: plain Markdown, no configuration syntax.
- H1: the name of the website or product.
- Blockquote: one to three sentences summarizing the offering.
- H2 sections: curated link lists, each link with one sentence of context (e.g. "Pages", "Docs", "Legal").
You can read a living example at flize.io/llms.txt: exactly this structure, updated with every new piece of content.
llms.txt vs. robots.txt vs. sitemap.xml
The three files are often confused but solve different problems:
- robots.txt controls access: which crawlers may read which paths?
- sitemap.xml lists exhaustively: all URLs, for systematic crawling.
- llms.txt curates: the most important content with context, so a language model can classify the site quickly and correctly.
In short: robots.txt says "may", sitemap.xml says "find", llms.txt says "understand".
Who needs llms.txt (and who does not)?
The file pays off most for websites whose offering needs explanation: SaaS products, documentation, agencies, shops with clear categories, specialist publications. In all of these, the machine’s first impression decides whether your content qualifies as a source. It adds little for pure image portfolios or single-page sites.
How do you create an llms.txt?
Four steps, realistically under an hour:
- 1. Collect core pages: the 10 to 30 pages that truly explain your offering.
- 2. One sentence per page: what does this page answer, and for whom?
- 3. Save as Markdown and publish at /llms.txt in the root.
- 4. Keep it current: every new relevant piece of content gets a line.
Is llms.txt actually being read?
The honest answer: there are no binding commitments from the big AI providers. llms.txt is an open community proposal (llmstxt.org) with growing adoption, especially in documentation ecosystems. Google’s John Mueller even compared the file to the long-meaningless keywords meta tag in April 2025: no AI service had confirmed using it, and server logs show hardly anyone requests it (Search Engine Journal, 2025). The math still stands: cost and risk are near zero, and every machine that does use the file gets clean context instead of guesswork. As part of a GEO setup it belongs to the basic kit.
Is llms.txt an official standard?
+
No. It is an open proposal by Jeremy Howard (Answer.AI) from September 2024, documented at llmstxt.org. Adoption is growing; official standardization is pending.
Does llms.txt replace robots.txt or sitemap.xml?
+
No, it complements both. Access control remains the job of robots.txt, and the complete URL list still comes from sitemap.xml.
What is llms-full.txt?
+
A widely adopted companion file that bundles the full content in one file instead of curated links, for systems that want to ingest a site in one go. It is not part of the llmstxt.org specification, which only describes the generated files llms-ctx.txt and llms-ctx-full.txt; it became a convention via documentation platforms. For most websites the normal llms.txt is enough.
How do I check my llms.txt?
+
Open the file in your browser at your-domain.com/llms.txt, check the Markdown syntax and the links. A GEO audit like the flize score checks the file automatically.
