Introduction

llms.txt is a proposed standard-a plain Markdown file at the root of your website (yourdomain.com/llms.txt) that gives large language models a curated summary of what your site contains and where to find key pages. Proposed by Jeremy Howard in September 2024, it strips away the HTML noise (nav, ads, scripts) that wastes AI context windows and replaces it with clean, structured content LLMs can read efficiently.

By late 2025, over 844,000 websites had adopted it—including Anthropic, Cloudflare, Stripe, and Cursor—and Google included it in its A2A protocol by May 2026. But there’s a catch: no major AI provider has formally committed to using it in production. It’s live on hundreds of thousands of sites, but the bots that matter barely touch it. Think of it as cheap insurance for an AI-first future, not a ranking hack for today.

What Does an llms.txt File Look Like?

The format is deliberately simple. You write it in Markdown. You place it at yourdomain.com/llms.txt. The spec defines a strict structure with four required elements in order:

  1. An H1 title with your project or site name. This is the only mandatory section.
  2. A blockquote containing a short summary of what the site does—key context the LLM needs before reading further.
  3. Zero or more markdown sections of any type except headings. Use paragraphs, lists, or any other Markdown to provide additional context about how to interpret the linked files.
  4. Zero or more H2 sections containing file lists. Each list is a series of markdown hyperlinks with optional descriptions after a colon.

Here’s a minimal example:

# My API Docs

> My API provides real-time weather data for over 200 countries.

This documentation is versioned. All examples assume v2 endpoints.

## Core Docs

- [Authentication](https://myapi.com/auth.md): OAuth2 setup and token management
- [Endpoints Reference](https://myapi.com/endpoints.md): Full list of REST endpoints
- [SDK Guide](https://myapi.com/sdk.md): Python and JavaScript SDK installation

## Optional

- [Changelog](https://myapi.com/changelog.md): Version history and deprecated features

The ## Optional section has special meaning. URLs listed here are secondary—LLMs can skip them when context window space is tight.

One important detail: the spec also recommends you provide .md versions of your HTML pages at the same URLs. If your docs page lives at /docs/quickstart.html, make a plain Markdown version available at /docs/quickstart.html.md. This gives LLMs a clean alternative to parsing your HTML.

llms.txt vs. llms-full.txt: What’s the Difference?

The ecosystem actually contains two companion files. Understanding the distinction prevents you from creating the wrong one.

llms.txt is the navigation index. It’s a curated list of your most important pages with one-line descriptions. Think of it as a table of contents for AI. It points to where the content lives without including the content itself. The file stays small—usually under 50 KB—making it fast to fetch and easy on context windows.

llms-full.txt is the full-content version. It embeds the actual text of every linked page into a single file. An AI system can ingest your entire documentation set with one request instead of crawling dozens of URLs. Anthropic’s llms-full.txt is 481,349 tokens. NVIDIA’s main site version runs 252,607 tokens.

Here’s when to use each:

FileBest ForTrade-off
llms.txtLarge documentation sites with many pages; situations where context windows are limitedRequires the LLM to make secondary requests to fetch actual page content
llms-full.txtSmaller sites where all content fits in an LLM’s context window; coding assistants that need complete API specs in one shotSingle large file; can be slow to fetch and expensive to process

Many sites publish both. Cloudflare organizes theirs by product (Workers, Pages, KV), with separate llms.txt and llms-full.txt files for each service. This modular approach lets AI tools request only the documentation relevant to a specific task.

How llms.txt Compares to robots.txt and Sitemaps

People call llms.txt “robots.txt for AI.” That comparison creates more confusion than clarity. These files serve fundamentally different purposes.

robots.txt controls access. It tells crawlers which paths they may and may not visit. It’s a gatekeeper. Major search engines and AI companies honor it. It has formal specifications (RFC 9309) and enforcement mechanisms.

sitemap.xml lists every indexable page on your site. It’s exhaustive, not curated. It helps search engines discover all your URLs. It says nothing about which pages matter most or how to interpret your content.

llms.txt curates context. It doesn’t block anything. It doesn’t list everything. It says: “Here’s what we do, here’s what’s important, and here’s where to find the details in formats you can actually read.”

Consider the practical difference:

  • A sitemap for a SaaS company might list 5,000 URLs. An LLM can’t process 5,000 pages of HTML in one context window.
  • An llms.txt file for that same company might list 12 critical docs pages with plain Markdown versions. The LLM gets exactly what it needs without the noise.

The files complement each other. Your llms.txt can even reference your sitemap.xml at the end for completeness. But using one doesn’t replace the need for the others.

Who’s Actually Using llms.txt (and Who Isn’t)

The adoption story is a split screen. On one side: widespread implementation by documentation platforms and developer tools. On the other: radio silence from major AI providers.

Who publishes llms.txt files:

  • Anthropic hosts both llms.txt (8,364 tokens) and llms-full.txt (481,349 tokens) for their Claude documentation
  • Cloudflare splits theirs by product—Workers, Pages, KV—each with dedicated AI-ready files
  • Stripe organizes by product categories with an “Optional” section for specialized tools
  • NVIDIA runs separate implementations for technical docs (1,259 tokens) and their main site (252,607 tokens)
  • Cursor, Vercel, Supabase, Zapier, Modal, and Coinbase all ship llms.txt files
  • Google created and maintains llms.txt files for Gemini Developer API docs, Chrome Developer Documentation, Firebase, and Flutter

Who officially supports reading them:

Nobody. At least not publicly.

  • OpenAI: GPTBot has been spotted hitting /llms.txt endpoints, suggesting testing or exploration. No official statement confirms ChatGPT uses the content during inference or citation.
  • Google: Gary Illyes stated plainly: “We currently have no plans to support LLMs.txt.” John Mueller compared it to the keywords meta tag on Reddit. Despite this, Google included llms.txt in its A2A protocol and Lighthouse audit—a confusing signal.
  • Anthropic: Claude’s team asked Mintlify to implement llms.txt for their own docs, but hasn’t confirmed Claude references these files during conversations.
  • Perplexity, Meta, Mistral: No statements, no documentation, no confirmed use.

Data backs up the skepticism. When Limy analyzed 515 million LLM bot traffic events, requests touching /llms.txt were statistically negligible across the user agents that actually drive citations—GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended.

SE Ranking ran a 300,000-domain study and found zero correlation between having an llms.txt file and getting cited more often by LLMs. Removing the llms.txt variable from their XGBoost model actually improved its accuracy.

Does llms.txt Actually Work? Separating Evidence from Hype

The honest answer is: maybe someday. Here’s what the data actually says.

The case for llms.txt:

  • Windsurf’s head of product engineering tweeted that llms.txt saves agents “time and tokens”
  • Vercel reports 10% of signups now come through ChatGPT—getting content into AI systems matters
  • Mintlify, which rolled out llms.txt to all hosted docs sites in November 2024, reports LLMs actively crawling these files
  • Anthropic specifically requested llms.txt and llms-full.txt implementation from Mintlify, suggesting real interest from at least one major AI lab
  • One case study from Springs Apps reported a 20% increase in search engine visibility and 15% improvement in accurate AI query answers after implementation

The case against llms.txt:

  • No major AI provider confirms using llms.txt in production inference or citation pipelines
  • Server log analysis shows minimal crawler traffic to /llms.txt endpoints from bots that matter
  • The 300,000-domain SE Ranking study found no statistical relationship between llms.txt presence and AI citation frequency
  • John Mueller’s keywords-meta-tag comparison stings because the keywords meta tag was widely adopted by website owners but ignored by the very engines it was built for

What this means for you:

llms.txt is a low-cost, no-downside bet. Creating one takes 20 minutes to an hour. It requires no ongoing maintenance beyond updating it when your core content changes. If the standard gets adopted, you’re positioned. If it doesn’t, you lost an afternoon.

What you shouldn’t do: expect llms.txt to recover traffic you’ve lost to AI search results. It’s not a ranking signal. It’s not a citation engine. It’s a structured invitation for AI systems to understand your site better—when and if they decide to accept it.

How to Create Your Own llms.txt File

You don’t need special tools. You need a text editor and a clear sense of what matters on your site. Here’s the process:

Step 1: Audit your content. Identify 5 to 20 pages that define what your site is about. Skip the blog archives. Skip the legal pages. Focus on product docs, API references, getting-started guides, pricing pages, and key landing pages.

Step 2: Write the Markdown file. Follow the spec structure. Start with your H1 title. Add a blockquote summary that answers “what is this site?” in two to three sentences. Add optional context paragraphs. Create H2 sections with markdown link lists.

Step 3: Create Markdown versions of linked pages. The spec recommends making .md versions available at the same URLs as your HTML pages. If you use a static site generator or docs platform, this is often automatic. nbdev projects generate .md versions by default. GitBook, Mintlify, Docusaurus, and VitePress have plugins that handle this.

Step 4: Place the file at your domain root. Upload llms.txt to yourdomain.com/llms.txt. Verify you can access it in a browser.

Step 5: Consider adding llms-full.txt. If your total documentation fits within a reasonable context window, create a concatenated full-content version. This is especially valuable for API docs where coding assistants need complete endpoint specifications in one fetch.

Step 6: Test it. Use the llms_txt2ctx command-line tool to expand your llms.txt into an LLM context file. Feed that context to ChatGPT or Claude and ask questions about your content. If the answers miss the mark, refine your descriptions and link selection.

Tools that can help:

  • Firecrawl’s llms.txt Generator: Crawls your site and generates compliant files automatically
  • llms.txt Checker: Validates your file against the spec
  • nbdev: Automatically creates .md versions of all pages for nbdev projects
  • Mintlify, GitBook, Fern, Publii, and other docs platforms: Built-in llms.txt generation

FAQ

Is llms.txt an official web standard?

No. It’s a community-driven proposal hosted on llmstxt.org with an open GitHub repository for discussion. It has no formal specification body like the IETF or W3C behind it. Think of it as an emerging convention, not a ratified standard.

Will llms.txt help my SEO?

Not directly. llms.txt targets AI inference—what happens when someone asks ChatGPT or Claude a question and the model fetches your content to answer. Traditional search engine ranking is a different mechanism. There’s no evidence that Google uses llms.txt as a ranking signal.

Does ChatGPT read my llms.txt file?

OpenAI hasn’t confirmed this. GPTBot has been observed requesting llms.txt files, which suggests experimentation, but there’s no documentation stating that ChatGPT’s browsing capability uses llms.txt content during responses.

Should I create an llms.txt file if I don’t have documentation?

Yes, if you care about how AI systems understand and represent your business. The spec works for any website—e-commerce stores can list product categories and policies, personal sites can outline CV information, and company sites can describe core offerings. You don’t need API docs to benefit from structured AI context.

How often should I update my llms.txt?

Update it when your core content changes. That typically means quarterly for stable documentation sites, or alongside every major product release. The file is small and manually curated, so don’t automate it into something that drifts from reality.

Can I use multiple llms.txt files?

The spec allows optional subpaths. You can place section-specific files at /docs/llms.txt, /api/llms.txt, or /guides/llms.txt. This modular approach helps AI tools fetch only the context relevant to a specific task.

Sources