What Is llms.txt? The File, the Format, and Whether It Works

llms.txt is a proposal by Jeremy Howard, co-founder of Answer.AI, published on September 3, 2024 at llmstxt.org. The reasoning: language models work with limited context windows and read cluttered HTML poorly, so a website could offer them 1 curated markdown file at /llms.txt, with a title, a short summary, and annotated links to the pages that matter most. A menu written for machines, sitting beside the robots.txt and sitemap.xml files websites already publish for crawlers.

Adoption since then is real, but concentrated. Documentation sites picked the file up fastest: Anthropic publishes one for its developer docs, Perplexity and Cloudflare do the same for theirs, and Mintlify, a documentation host, generates the file by default for the sites it serves. Cruelx publishes its own at cruelx.com/llms.txt, and the directories tracking the format list hundreds of sites. What the list is short on is confirmation from the consuming side, and that is where this article is headed.

The format is deliberately small. It is a markdown file served as plain text at your domain root, and only 1 element is required. The table below is the complete anatomy; everything after the H1 is optional.

A complete llms.txt example

Here is the whole format in use, for a fictional dental clinic. This is a valid, complete file: nothing in the spec asks for more.

# Northgate Dental

> Northgate Dental is a family dental clinic in Portland, Oregon, offering preventive care, implants, and same-day emergency appointments.

Patients book online. Fees and accepted insurance plans are published on the pricing page.

## Pages

- [Services](https://www.northgatedental.com/services): Every treatment offered, with fees and recovery times
- [Pricing](https://www.northgatedental.com/pricing): Standard fees and accepted insurance plans
- [Book an appointment](https://www.northgatedental.com/book): Online booking, including same-day emergency slots

## Optional

- [Our team](https://www.northgatedental.com/team): Dentist bios and credentials
- [Blog](https://www.northgatedental.com/blog): Dental health guides and patient FAQs

• The H1 and the blockquote do most of the work. A model that reads nothing else still gets the business name, the category, and the location.
• Every link carries a 1-line description after the colon, so a system can choose the right page without fetching all of them.
• The Optional section holds the links a model can skip. The spec defines it as the part to drop when context runs short.
• The whole file is genuine markdown, which keeps it readable for any human who opens it too.

You can write the file by hand in any text editor, or use the free llms.txt generator and validator: it builds the file from your details and checks an existing file against the format, including the parts sites most often get wrong.

What is llms-full.txt?

The same proposal defines a companion file, llms-full.txt, which inlines the full text of your pages in 1 markdown document instead of linking to them. Documentation sites use it so a model can load an entire docs set in a single fetch; Anthropic’s runs to thousands of lines. For a typical business site it is overkill: the linked summary covers the use case, and a full-text mirror is 1 more thing to keep current. The spec also suggests offering markdown versions of individual pages by appending .md to their URLs; outside documentation tooling, almost nobody does.

Here is the honest answer: there is no evidence that llms.txt improves your visibility in AI answers, and no major engine has confirmed consuming the file. OpenAI, Google, and Anthropic all document their crawlers and their controls in detail, and none of that documentation mentions reading llms.txt. Google is the most explicit of the 3: its guidance on AI features in Search, updated in 2025, lists AI-specific text files among the things sites do not need for its AI features.

Google’s search relations team has been blunter still in public comments, comparing the file to the keywords meta tag: a place where sites describe themselves, which ranking systems learned long ago not to trust. Site owners who publish the file and watch their server logs report the same pattern from the outside: the major AI crawlers fetch pages, robots.txt, and sitemaps, and mostly leave /llms.txt alone. As of August 2026, publishing the file is a signal sent into a room where nobody has confirmed listening.

So why publish one at all? 3 modest reasons survive the evidence. User-directed AI agents (a model sent to your site by a person, a coding assistant, a research agent) can read the file when they arrive, and a curated summary gives them a better first page than your navigation. The cost is close to zero: 1 static text file, no code, no risk to anything that already works. And the exercise itself has value, because writing a 3-sentence summary of your business and choosing your 5 most useful pages is clarifying in the way good positioning work always is.

The work that measurably moves AI visibility lives elsewhere: crawlable rendered HTML, answer-first pages, third-party mentions, and the Google rankings AI engines retrieve from. Our guide to getting recommended by ChatGPT ranks those factors by evidence. And measurement beats faith in both directions: the Cruelx AI Visibility Scan asks 6 AI engines the questions your buyers ask and reports, engine by engine, whether you appear, for $19.99 one time.

llms.txt is often described as robots.txt for AI, and the comparison is exactly wrong: robots.txt controls access, while llms.txt controls nothing. The table separates the 3 files that live at the root of a site.

The practical consequence: if your question is which AI systems may access your site, the answer is in robots.txt. llms.txt cannot allow or block anything. It can only describe.

Because access lives in robots.txt, the decisions that matter are made per crawler, and the names are worth knowing, because each serves a different purpose. Blocking a training bot does not remove you from AI search, and blocking a search bot does not keep your content out of training. These are the robots.txt user-agent tokens behind the major AI surfaces.

Notice what the table does not contain: none of these crawlers documents llms.txt as part of its behavior, and all of them document robots.txt. Access is decided in 1 file, and it is not the new one. The free AI visibility checker reads your robots.txt and shows the verdict per crawler, and it checks whether your llms.txt is present and readable while it is there.

Yes, on the smallest possible terms: because it is 10 minutes, free, and harmless, not because it will change your numbers. Here is the whole job.

1. 1Write the summary.1 H1 line with your business name, then a blockquote of 2–3 sentences saying what you do, who you serve, and where. This is the part a model is most likely to use if it reads the file at all.
2. 2Choose the links.5–15 pages that genuinely answer buyer questions: services, pricing, key guides, contact. 1 line of description per link. Skip the rest of your sitemap; curation is the point.
3. 3Generate and validate. The free llms.txt generator produces the file in the correct format, and validates a file you already have against the spec.
4. 4Serve it at the root. The file belongs at yourdomain.com/llms.txt as plain text. No redirect chains, no HTML wrapper.
5. 5Then spend your energy up the stack. The measured levers for AI visibility are your content, your crawlability, and your third-party footprint. Our guide to making your website easier for AI to understand covers them in order.

That is the honest state of llms.txt in August 2026: a tidy idea, genuinely adopted on the publishing side, unconfirmed on the reading side, and cheap enough that none of that needs to stop you. Publish the file, spend the 10 minutes, and then put your attention where the evidence is.