
ℹ️ Geodeck is built by the team behind Seofable, an AI-era SEO content tool. Seofable is listed in our directory as a clearly labeled featured listing; rankings and recommendations in this article are editorial.
What Is llms.txt (and What It Isn't)
llms.txt is a plain-text file written in Markdown, placed at the root of a domain, that gives AI systems a short summary of a site and a curated list of links to its most important pages. It's a proposal, not a standard anyone has ratified. No browser enforces it, no search engine requires it, and no AI lab has committed in writing to reading it on a schedule.
That last point matters because the name invites confusion with robots.txt. robots.txt is an access-control file: it tells well-behaved crawlers what they may and may not request. llms.txt does nothing of the sort. It can't block a crawler, can't deindex a page, and can't force an AI model to prioritize anything. Think of it closer to a curated table of contents left on a server, available to anyone who happens to look for it.
Who proposed the standard and why
Jeremy Howard, cofounder of Answer.AI and fast.ai, published the llms.txt specification at llmstxt.org in September 2024. His reasoning was practical: large language models have a finite token context window, and most websites are too cluttered with navigation, ads, and scripts for a model to extract the useful parts efficiently. A short, hand-curated Markdown file, the argument goes, saves the model from having to parse an entire rendered page just to find the three paragraphs that actually matter.
What a well-formed llms.txt file looks like
A compliant file opens with an H1 title (usually the site or product name), followed by a one-line blockquote summary, then a series of H2 sections grouping links by topic. Each link is a standard Markdown link with an optional short description after a colon or dash separator. There's no required length, but the spec's own examples run well under 2,000 words, because the whole point is brevity, not completeness.
llms.txt vs robots.txt vs sitemap.xml vs llms-full.txt
These four files solve four different problems, and confusing them is the single most common mistake site owners make.
| File | Purpose | Enforceable? | Who typically reads it | Format |
|---|---|---|---|---|
| robots.txt | Tells crawlers what they may request | Yes, by convention, most compliant bots honor it | Search engine bots, some AI crawlers (GPTBot, ClaudeBot) | Plain text, directive syntax |
| sitemap.xml | Lists URLs for discovery and indexing | No, it's a hint not a rule | Search engines (Google, Bing) | XML |
| llms.txt | Curated summary and link list for AI context | No | Unconfirmed, optional for any AI system | Markdown |
| llms-full.txt | Full page content inlined, not just links | No | Unconfirmed, optional | Markdown, often very large |
The llms-full.txt variant exists because llms.txt alone only points to pages, it doesn't contain them. llms-full.txt pastes the actual content of every linked page into one file, so a model theoretically never has to fetch anything else. The problem is scale. A documentation site with 300 pages at 1,500 words each produces a file well north of 400,000 words. Most model context windows, even generous ones, can't hold that in a single pass, and Ahrefs' breakdown of llms.txt flags this exact token ceiling as the practical limit on llms-full.txt for anything beyond a small site.
Does llms.txt Actually Influence AI Search Results? The Evidence So Far
The honest answer is: nobody has published proof that it does, and Google has explicitly said it doesn't matter for its own ranking or AI features. That's a meaningful gap between what's been proposed and what's been confirmed.
What Google has said
Google's position has moved from informal to official. On 15 June 2026, Google added a section to its AI optimization guide, headed "Clarifying guidance on llms.txt files," stating that the file is not needed to appear in Google Search and that keeping one will neither harm nor help a site's visibility or rankings because Google Search ignores it. Google's own wording puts it directly: creating one will "neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." This followed Google's Gary Illyes, who, speaking at Google's Search Central Deep Dive event in Asia-Pacific on 23 July 2025, said Google "doesn't support llms.txt and isn't planning to," and that ordinary SEO, not a separate llms.txt file, is what drives visibility in AI Overviews. Search Engine Land's coverage makes a similar distinction: llms.txt behaves more like an optional map left for anyone willing to look, not a signal plugged into ranking. If a page isn't crawled, rendered, and indexed through normal means, an entry in llms.txt doesn't fix that.
What ChatGPT, Perplexity, and Gemini actually do with it today
None of OpenAI, Anthropic, Perplexity, or Google has published documentation confirming that their crawlers or retrieval systems automatically fetch and parse llms.txt files. Their crawlers, GPTBot and OAI-SearchBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity, Google-Extended for Google, are documented to respect robots.txt directives. Whether any of them specifically request /llms.txt as part of a routine crawl is not documented anywhere by the vendors themselves.
Community testing reflects the same uncertainty. A widely discussed Reddit thread in r/SEO collects site owners reporting that the file sits unread in server logs for weeks after publishing. SE Ranking's own analysis goes further, arguing in its llms.txt piece that brands adopted the file faster than any AI platform adopted reading it. Treat llms.txt as speculative infrastructure right now, worth having for other reasons covered below, but not a confirmed lever for AI citations.
How to Create an llms.txt File (Step-by-Step)
Building one takes under an hour for a site with a clear content structure. The steps are mechanical:
- Write a one or two sentence summary of what the site does, in plain language, no marketing copy.
- List your 10 to 30 most important pages, grouped by topic, not by every URL that exists.
- Format each section as an H2, each link as a standard Markdown link with a short description.
- Keep the whole file under roughly 2,000 words. If you're inlining content instead of linking to it, that belongs in llms-full.txt, not here.
- Upload it to the domain root, so it resolves at `https://example.com/llms.txt`, exactly like robots.txt.
- Re-check it whenever you publish or retire major pages. A stale llms.txt pointing at dead URLs is worse than no file at all.
Minimum viable llms.txt template
```markdown
Example Co builds project management software for construction teams.
Product
Documentation
- Getting started: Setup guide for new accounts
- API reference: Endpoints and authentication
Company
```
Common mistakes: bloated files, broken links, stale content
Three things break an otherwise fine llms.txt. Files that exceed 5,000 words defeat the purpose, a model still has to parse all of it to find the relevant two lines. Dead links accumulate fast on sites that redesign URLs without updating this one file nobody remembers exists. And nobody assigns ownership, so the file gets written once during a GEO sprint and never touched again, even as the pages it points to change or disappear.
Tools That Generate llms.txt Automatically
Manual generation works fine for small sites, but CMS plugins and dedicated GEO platforms now automate most of it. Rank Math added an llms.txt module that builds the file from existing WordPress post and page data, with settings to control which post types and taxonomies are included. AIOSEO ships a comparable feature, generating a file directly from a site's existing structure rather than asking you to hand-curate links.
Beyond WordPress plugins, a growing set of dedicated GEO tools bundle llms.txt generation with broader AI-visibility features, like content scoring and crawlability checks. Geodeck's GEO tools directory compares these by feature set and pricing rather than listing every vendor that ships a generator button. Evaluation here comes down to three things: does the tool keep the file synced automatically as content changes, does it validate the Markdown output against the llmstxt.org spec, and is generation bundled into a broader GEO workflow or sold as a standalone gimmick. Some platforms go further than file generation entirely. Scrunch AI, for instance, builds an Agent Experience layer aimed at how AI crawlers interact with a site, treating llms.txt as one small piece rather than the whole strategy. Otterly AI is useful for running crawlability checks alongside llms.txt implementation, to confirm the file is actually reachable and well-formed before assuming anything reads it.
How to Verify Your llms.txt Is Actually Being Used
Verification is the step almost every article on this topic skips, and it's the only way to know if the file is doing anything.
Checking server logs for AI bot hits
Filter raw access logs for known AI crawler user-agents, GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended, Applebot-Extended, and look specifically for requests to `/llms.txt`. Most hosting platforms and CDNs (Cloudflare, Vercel, Netlify) expose log search for this without needing raw shell access.
Take a hypothetical documentation site with 200 pages that publishes llms.txt in January. Filtering 30 days of logs afterward for GPTBot and PerplexityBot shows both bots requesting `/robots.txt` and `/sitemap.xml` daily, as expected. Requests to `/llms.txt`, by contrast, show up only a handful of times across the month, with no regular pattern, consistent with manual curiosity-driven checks rather than a scheduled crawl. That pattern, or the absence of any hits at all, is the clearest signal available right now about real-world adoption.
Tracking brand mentions in ChatGPT, Perplexity, and Gemini over time
Run the same set of prompts against ChatGPT, Perplexity, and Gemini before publishing llms.txt, then again on a fixed schedule afterward, weekly works better than daily since retrieval indexes lag behind live content. Record whether your brand gets cited, whether the cited URL matches what's in your llms.txt, and whether that changes over four to eight weeks. A single before/after snapshot isn't enough; AI answers vary run to run even with no site changes at all.
Worth noting here: coverage across monitoring tools is uneven. Across a hand-verified dataset of 64 GEO tools, only 48% track Google's AI Overviews while 68% track Perplexity, meaning the category is built to watch chatbots more than the AI layer sitting inside Google Search itself (source, as of August 2026). If you're trying to verify llms.txt's effect specifically on Google's AI features, confirm your monitoring tool actually covers AI Overviews before trusting a flat line as a null result. Geodeck's AI visibility monitoring resources cover how to track these citation changes across engines without assuming one platform's behavior represents all of them.
Honest Expectations: When llms.txt Helps and When It's a Waste of Time
llms.txt is not a substitute for content quality, schema markup, or the editorial signals that make a page trustworthy to begin with. If a page isn't well-written or well-structured on its own, listing its URL in a curated file doesn't fix that.
| Site type | Realistic effect of llms.txt |
|---|---|
| Small brochure site, under 20 pages | Negligible. Pages are already simple enough for live crawling to handle without help. |
| Large documentation or SaaS knowledge base | Modest at best. May reduce ambiguity about which pages are canonical, assuming anything reads the file. |
| Ecommerce catalog with thousands of SKUs | Minimal. llms.txt can't list every product without becoming llms-full.txt-sized, and that hits token limits fast. |
| News or publisher with fast-changing content | Risky. Links go stale quickly; a file updated monthly on a site that publishes daily misleads more than it helps. |
Skip llms.txt if your dev time is better spent on schema markup, page speed, or fixing thin content, all of which have documented effects on both traditional SEO and how AI systems represent a page. Build one if you've already handled those basics and have 30 minutes to spare. It's low-risk, low-cost, and positioned correctly within a broader GEO and AEO strategy, it's a small addition, not a centerpiece.
FAQ
Is llms.txt a Google ranking factor?
No. Google updated its official guidance on 15 June 2026 to state explicitly that Search, including AI Overviews and AI Mode, ignores llms.txt files, and that having one will neither help nor hurt rankings.
Do ChatGPT, Perplexity, and Gemini actually read llms.txt?
No major AI platform has publicly confirmed automatic parsing of llms.txt as of now. Treat it as speculative and early-stage, not as a guaranteed input to any model's output.
What's the difference between llms.txt and llms-full.txt?
llms.txt is a short summary with curated links to key pages. llms-full.txt inlines the full content of those pages directly into the file, which can hit token and context-window limits fast on any site larger than a few dozen pages.
Where should I place my llms.txt file?
At the site root, the same convention robots.txt uses: `https://example.com/llms.txt`.
Can llms.txt replace robots.txt or my sitemap?
No. robots.txt controls crawl permissions, sitemap.xml aids indexing and discovery, and llms.txt curates content summaries for AI systems. They serve different purposes and belong alongside each other, not as substitutes.
How do I know if my llms.txt file is actually working?
Check server logs for requests from AI crawler user-agents like GPTBot, PerplexityBot, and ClaudeBot specifically to `/llms.txt`, and use AI visibility monitoring to track whether brand citations in ChatGPT, Perplexity, and Gemini shift over four to eight weeks after publishing the file.
Fact-checked against live sources, 2026-10-08: Verified Jeremy Howard/Answer.AI/fast.ai authorship and September 2024 publication date (accurate); corrected the article's framing of Google's position — updated from "hasn't confirmed" to reflect Google's explicit 15 June 2026 guidance update and Gary Illyes' July 2025 statement that Google ignores llms.txt for Search and AI Overviews/AI Mode; verified Rank Math's llms.txt module exists but softened the unverified "auto-regenerates on content change" claim; crawler user-agent names (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended) confirmed accurate..