Introduction

You can’t improve what you don’t measure. An AI readability audit shows you exactly what a language model sees when it lands on your site. Before you write an llms.txt file — or restructure your docs for AI consumption — you need to know what an LLM actually encounters. That picture is often uglier than you think.

What AI Readability Actually Means

Traditional readability scores measure human comprehension—Flesch-Kincaid grade levels, sentence length, word complexity. AI readability is different. It measures how efficiently a language model can locate, extract, and interpret your content given a fixed context window.

Three factors determine AI readability:

  • Signal-to-noise ratio. How much of the fetched content is useful information versus structural markup. A page with 2,000 words of content embedded in 8,000 words of HTML has a 1:4 ratio. Bad.
  • Content discoverability. Can the AI find your key pages without crawling every link on your site? If your architecture buries API docs under six layers of navigation, the answer is no.
  • Format compatibility. LLMs process Markdown, plain text, and clean HTML efficiently. They struggle with JavaScript-rendered content, iframe-embedded docs, and image-based text.

These aren’t hypothetical concerns. When Windsurf’s engineering team analyzed agent crawling patterns, they found that sites with clean content architecture consumed 60% fewer tokens per successful information retrieval—which directly translates to faster, cheaper, and more accurate AI responses that reference your content.

Step 1: Simulate What an LLM Sees

Start by stripping away everything that isn’t content. You have two approaches.

The manual approach. Open your five most important pages. Use a browser’s “View Source” or a tool like curl to fetch the raw HTML. Copy the full output into a character counter. Then copy just the visible text content. Divide content characters by total characters. If the ratio sits below 30%, you have a signal-to-noise problem.

The automated approach. Use an LLM fetcher simulator. Firecrawl’s llms.txt generator crawls your site and produces a clean content report. The llms_txt2ctx CLI tool expands a context file and shows you exactly what the model receives. Run your top 20 pages through either tool and read the output yourself. If it confuses you, it confuses the model.

Look for these specific failure modes:

  • Navigation menus occupying the first 500+ words of output
  • Cookie banners and newsletter popups injected into the content stream
  • JavaScript-dependent content that doesn’t appear in the raw fetch (SPAs, dynamic renders)
  • Repeated footer text that appears identically on every page and wastes tokens
  • Inline CSS and script blocks adding thousands of meaningless characters

Document what you find. You’ll fix the worst offenders in step 4.

Step 2: Map Your Content Topology

An LLM doesn’t browse your site like a human. It sends requests to specific URLs. Your job is making sure the right URLs return the right content in the right format.

Start by listing every page an AI should know about. This is a different list than your sitemap. Your sitemap includes everything. Your AI map includes only what defines your business, product, or domain.

Ask these questions:

  • If an AI could only fetch five pages from your site, which five would make it an accurate source?
  • Which pages contain information that answers the ten most common questions about your product or topic?
  • Where does your unique data live? Pricing tables, API specifications, research findings, product comparisons?
  • Which pages do citation-worthy content? An LLM cites sources that contain definitive statements, not marketing fluff.

Now check whether these pages exist as clean HTML or—ideally—as Markdown. If your key pricing page renders entirely through JavaScript, the AI gets nothing. If your API reference lives in a PDF, the AI needs OCR. Flag every format mismatch.

Finally, examine your internal link structure. Can an AI navigate from your homepage to every critical page in three clicks or fewer? If not, your information architecture needs work. LLMs don’t fill out search forms or use hamburger menus.

Step 3: Audit Your Existing AI Crawler Access

You need to know which AI bots already crawl your site and what they’re doing. This data tells you whether your current setup helps or hinders them.

Check your server logs. Search for requests from these user agents:

  • GPTBot (OpenAI)
  • Claude-Web and ClaudeBot (Anthropic)
  • PerplexityBot (Perplexity)
  • Google-Extended (Google’s AI products)
  • OAI-SearchBot (OpenAI’s search feature)
  • Applebot-Extended (Apple Intelligence)

Look at what they request, what status codes they get, and how many bytes you serve them. If GPTBot requests your docs page and gets a 301 redirect through three URLs before landing, you’re burning its token budget on redirects.

Verify your robots.txt. Many companies block AI crawlers entirely, then wonder why they don’t appear in AI-generated answers. Check your robots.txt for directives targeting the user agents above. There’s a legitimate debate about whether to allow them. But if you want AI visibility, blocking the crawlers guarantees you won’t get it.

Count your Markdown endpoints. Tally how many of your critical pages have .md equivalents accessible at the same URL pattern. If the answer is zero, that’s your biggest gap. The llms.txt spec recommends serving Markdown versions at URLs like /docs/api.md alongside /docs/api.html. Without these, every AI request pays the HTML parsing tax.

Step 4: Prioritize and Fix

Turn your audit findings into a ranked action list. Not everything matters equally. Here’s the order:

Critical (fix immediately):

  • Pages that return empty or garbled content to LLM user agents due to JavaScript rendering, authentication walls, or bot-blocking
  • Missing Markdown versions of your five most important pages
  • A robots.txt that blocks AI crawlers from your core content while allowing them everywhere else

High priority (fix this week):

  • Navigation and footer markup dominating the first 1,000 tokens of every page fetch
  • Key pages orphaned more than three clicks from any crawlable entry point
  • Content embedded in PDFs, videos, or images with no text equivalent

Medium priority (fix this month):

  • Creating an llms.txt file that points to your audited and cleaned pages
  • Building an llms-full.txt if your total critical content fits within current context windows (typically 128K-200K tokens)
  • Adding structured summaries to long-form pages so AI systems can determine relevance before deep reading

Low priority (do when convenient):

  • Converting legacy blog content to AI-readable formats
  • Adding metadata to secondary pages
  • Creating subpath-specific llms.txt files for different product areas

Each fix should include a verification step. After making changes, re-run your manual or automated fetch test. Compare before and after token efficiency. If your top page went from 25% content-to-markup ratio to 70%, you’ve done the work.

How to Test Your Results

Auditing without testing is performative. You need to verify that your changes actually improve AI comprehension.

The direct test. Feed your cleaned content into ChatGPT, Claude, or Perplexity and ask questions that your site should answer. Use specific prompts like:

  • “What are the main pricing tiers for [your product] based on this content?”
  • “What’s the first step to integrate [your API] into a Python application?”
  • “Summarize what [your company] does in two sentences using only the provided information.”

If the model hallucinates, confuses your product with a competitor, or says it doesn’t have enough information, your content still has readability problems.

The comparison test. Take your best-performing page and run two versions through the same prompt: the raw HTML version and your cleaned Markdown version. Time both responses. Measure accuracy. The Markdown version should be faster and more accurate. If it isn’t, the problem isn’t format—it’s content quality.

The monitoring setup. Configure your analytics or logging to track AI bot activity patterns before and after your audit. Set up alerts for 404s and 500s returned to AI user agents. Watch for increases in successful fetches to your cleaned content URLs. If AI bots start spending more time on your actual content pages and less time crawling navigation, your audit worked.

FAQ

How long does an AI readability audit take?

For a site with under 50 pages, the full audit takes two to three hours including testing. Larger documentation sites might need a day. The longest step is usually creating Markdown versions of key pages if you don’t have automated tooling for that.

Do I need developer help to run this audit?

Server log analysis typically requires developer or DevOps access. Everything else—fetching pages, counting content ratios, testing with LLMs—can be done by a technically-minded content or SEO person.

What if my site is a single-page application?

SPAs are the hardest case. LLM fetchers don’t execute JavaScript. Your content is invisible to them. You need server-side rendering, prerendering, or static Markdown versions of every important page. This is a significant engineering investment, but it’s the only way AI systems can access SPA content.

Should I remove my cookie banner and newsletter popups for AI crawlers?

You can’t selectively hide them for bots while showing them to humans without server-side logic. But you can serve different content to known AI user agents. Several companies do this. Just don’t cloak—make sure the information AI crawlers get matches what humans see.

How do I know if my audit actually helped?

Track three metrics: AI bot successful-fetch volume to your content pages, citation frequency in AI-generated answers (manual spot-checking works), and the token efficiency ratio of your top pages over time. Improvement in all three confirms your audit had impact.

Sources