llms.txt is a proposed plain-text file, published at the root of a website, that gives AI systems and AI agents a short, curated index of a site's most important pages in Markdown. Google has stated directly that Search ignores the file, and no major AI assistant has confirmed that it reads llms.txt at query time, so it is not a ranking or citation lever. It can still be a useful convention for documentation-heavy sites and for AI agents that browse the live web, and this guide draws that line precisely.
What Is llms.txt?
llms.txt takes its name from a proposal that Jeremy Howard, co-founder of the AI research lab Answer.AI, published on September 3, 2024 at llmstxt.org. His rationale is straightforward: HTML pages are built for people, wrapped in navigation, ads, and JavaScript that make them slow and error-prone for a language model to parse, and even a clean page can be too long for a limited context window. An llms.txt file at a site's root gives an AI agent a short, curated map instead, listing the pages worth fetching and skipping the rest.
The important word is proposal. llms.txt has no standards body behind it, no IETF or W3C process, and no browser or search engine that treats it as a required input. That does not make it worthless, but it does mean every claim about what llms.txt does needs a source, which is the approach this guide takes throughout.
What Does an llms.txt File Look Like?
The specification defines a narrow Markdown structure, most of it optional:
- An optional byte-order mark, which most site owners can ignore.
- An H1 with the name of the site or project. This is the only required section.
- A blockquote directly under the H1 with a one- or two-sentence summary and any fact an agent needs before reading further.
- Zero or more paragraphs or lists of plain context, with no headings inside them.
- Zero or more sections marked by H2 headings, each a Markdown list of links in the form
[name](url): optional note.
By convention, one H2 section is named "Optional" and holds secondary links an agent can skip when it needs a shorter context window. Example: a developer-tools company with a documentation site might structure its file like this.
# Acme Docs
> Acme Docs covers the Acme API and SDKs: authentication, webhooks and the core REST endpoints, aimed at developers integrating Acme into their own product.
## Docs
- [Getting started](https://docs.acme.dev/getting-started): account setup, API keys and your first request.
- [API reference](https://docs.acme.dev/api-reference): every endpoint, parameter and response shape.
## Guides
- [Webhooks](https://docs.acme.dev/guides/webhooks): event types, retries and signature verification.
## Optional
- [Changelog](https://docs.acme.dev/changelog): release notes by version.
Every link needs real anchor text and a working URL. Notes after the colon are optional, but they are what actually helps an agent decide whether to fetch a page, so treat them like a one-line abstract rather than a label.
What Is llms-full.txt?
llms-full.txt is not part of the core specification published on llmstxt.org, which makes no mention of it. It emerged as a convention among early adopters and the tooling Howard built alongside the proposal: instead of linking out to each page, llms-full.txt concatenates the full text of every listed page into one long Markdown file, so an agent can read everything in a single fetch.
The trade-off is context budget against fetch latency. llms.txt costs an agent almost nothing to read and lets it decide what to fetch next; llms-full.txt costs more tokens up front but removes the need for follow-up requests. Documentation platforms with a few hundred pages, such as API references, publish llms-full.txt most often, since the whole corpus can still fit inside a large context window.
Example: a documentation site with 40 reference pages averaging 800 words each. An llms.txt index listing every page with a one-line description runs about 600 words, roughly 800 tokens at a typical English word-to-token ratio of 1.3, cheap enough for an agent to fetch every session. The equivalent llms-full.txt, with full page text concatenated, runs past 30,000 words, tens of thousands of tokens, which only a large context window can absorb in one request. Assumptions: illustrative word counts and a standard tokenizer ratio; actual figures depend on your content and the model reading it.
llms.txt vs robots.txt vs sitemap.xml: What's the Difference?
The three files are easy to confuse because they all sit in a site's root as plain text, but they solve different problems for different audiences. robots.txt has existed since 1994 and was formalized by the Internet Engineering Task Force as RFC 9309 in September 2022; it tells automated crawlers which URLs they may access, and Google is explicit that it is not a mechanism for keeping pages out of Search. sitemap.xml, standardized through the sitemaps.org protocol, is a comprehensive list of a site's indexable URLs that search engines crawl on their own schedule.
llms.txt, by contrast, is read on demand, when an agent is already assembling context for a specific task, and it is curated rather than exhaustive by design.
| File | Governing body | Audience | Read when | Exhaustive? | | --- | --- | --- | --- | --- | | robots.txt | IETF, RFC 9309 (2022) | Crawlers of all kinds | Before a crawl, to check permission | No, it lists rules, not pages | | sitemap.xml | sitemaps.org protocol | Search engine indexers | On the engine's own crawl schedule | Yes, every indexable URL | | llms.txt | No standards body; a single proposal | AI agents and assistants | On demand, when an agent needs site context | No, curated by the site owner |
Does llms.txt Actually Help You Get Cited by AI?
The honest answer, as of late 2026, is that llms.txt has not been shown to influence rankings, AI Overviews, or chatbot citations, and the companies best positioned to know have said so directly. Google's developer documentation on optimizing for generative AI features states that sites do not need new machine-readable files, AI text files, or Markdown to appear in Search or its generative features, and that adding one will neither help nor hurt a site's visibility or rankings, because Google Search ignores them. The same page adds that maintaining a file for other tools that do use it is fine, it is simply not necessary for Google's own systems.
Google's Search Advocate John Mueller has gone further in public comments, describing any AI-visibility benefit from llms.txt as speculative and pointing out that although the file has existed for years, no AI system is known to require or rely on it. Neither OpenAI, Anthropic, nor Perplexity has published documentation stating that its assistant or crawler reads llms.txt when answering a query or indexing a page. That silence is itself informative: companies that do rely on a specific mechanism for grounding answers, such as retrieval over a search index, tend to document it clearly.
One nuance is worth separating out. Chrome added an llms.txt check to Lighthouse's Agentic Browsing audits in 2026. That audit only confirms a file is present and parses as Markdown, not that it changes visibility anywhere, and a missing file is marked "Not Applicable" rather than failed, because providing one is explicitly optional. Lighthouse is checking whether a site is ready for a future where people delegate browsing to AI agents, a separate question from whether llms.txt earns a citation today.
Illustrative scenario: a documentation site adds an llms.txt file and, a few weeks later, notices more ChatGPT citations. Before crediting the file, check whether the same window saw a new integration announcement, a Product Hunt launch, or new backlinks from developer communities. Attributing a citation increase to llms.txt without ruling out the more likely causes is one of the most common mistakes in early GEO reporting.
When Is Building an llms.txt File Worth It?
llms.txt earns its cost on sites where the content is dense, technical, and genuinely consumed by AI agents doing real work: API references, SDK documentation, CLI tools, and developer platforms where an agent such as a coding assistant is often the actual reader, not a person browsing in a tab. It earns very little on a marketing site, a blog, or an e-commerce catalog, where the AI systems that matter already rely on the same crawled index and structured data that power traditional search.
Run through this checklist before investing engineering time:
- Does an AI coding agent or assistant plausibly need to browse your docs live, rather than answer from training data or a search index?
- Is your content currently hard for a fast, low-context fetch to parse: heavy client-side rendering, sparse headings, marketing copy mixed with reference material?
- Do you already have clean, canonical URLs and short descriptions for each page, the raw material an llms.txt file needs?
- Can the file be generated automatically when docs change, instead of maintained by hand?
- Have you already covered the fundamentals that matter to every AI system, including the ones that ignore llms.txt entirely: indexable HTML, structured data, and clear entity signals?
If you answered yes to the first three and can satisfy the fourth, building an llms.txt file is a low-cost addition. If you are reaching for it as a fix for weak AI visibility elsewhere, redirect that budget to the fundamentals covered later in this guide instead.
How Do You Implement llms.txt?
Implementation is simple by design, which is part of why it spread faster than its evidence base. Two approaches cover almost every site.
A static file. For a site with a stable structure, hand-write llms.txt and place it in your public directory (public/llms.txt in Next.js) so it serves at /llms.txt. Update it whenever you add or retire a major section.
A generated route. For a site whose content changes often, generate the file from the same source your pages already use, so it never drifts out of date. In the Next.js App Router, a route handler can build it from your content collection at request time or build time:
// app/llms.txt/route.ts
import { getAllPosts } from "@/lib/posts";
const siteUrl = process.env.NEXT_PUBLIC_SITE_URL ?? "https://example.com";
export async function GET() {
const posts = await getAllPosts();
const guides = posts.filter(
(post) => post.category === "GEO" || post.category === "SEO"
);
const lines = [
"# Example Co",
"",
"> Example Co's technical guides and product documentation.",
"",
"## Guides",
...guides.map(
(post) => `- [${post.title}](${siteUrl}/blog/${post.slug}): ${post.description}`
),
];
return new Response(lines.join("\n"), {
headers: { "content-type": "text/markdown; charset=utf-8" },
});
}
This keeps the file honest: if a post's title or description changes, or a page is removed, the output updates on the next build without anyone remembering to edit a static file by hand.
Some documentation platforms go a step further and serve a Markdown mirror of every page, often at the same URL with .md appended, so an agent that lands on any page, not just the root file, can fetch clean Markdown instead of parsing rendered HTML. Both ideas address the same root cause: crawlers and agents parse markup far less reliably than people do.
What Matters More Than llms.txt for AI Visibility?
llms.txt addresses a narrow problem: page parsing for an agent already on your site. It does nothing for the harder problem most companies actually have, which is getting found and cited in the first place. Four levers matter more, roughly in the order to tackle them.
- Crawlable, server-rendered content. If an AI crawler cannot fetch your page's text without executing JavaScript, no root-level file fixes that. Our JavaScript SEO guide for React and Next.js covers the rendering strategies that keep content visible to both search engines and AI crawlers.
- Structured, entity-clear content. Schema markup and consistent facts about your organization help every system, not only the ones that happen to read a proprietary file. See our structured data and schema markup guide and our guide to entity SEO and the knowledge graph.
- Topical authority and third-party citations. Generative engines lean on authority signals built from being cited by other credible sites, not from a file only you control. That is the premise of generative engine optimization as a discipline.
- Measurement. You cannot know whether any of this works without tracking citations and mentions across the AI systems your buyers actually use.
If you are evaluating a vendor that leads its pitch with llms.txt as a flagship deliverable, treat that as a signal worth noticing on its own. Our guide to choosing a GEO agency lists the questions that separate real technical work from a file nobody reads.
How Agentixly Approaches AI Visibility and Technical GEO
When a client asks whether they need an llms.txt file, we treat it as one item on a technical GEO checklist, not the headline deliverable. A typical engagement inside our GEO service runs in four phases.
- Crawler and rendering audit. We fetch key pages the way AI crawlers do, compare rendered output to raw HTML, and fix any gap that hides content from common AI crawlers.
- Structured data and entity work. We implement or repair Organization, Article, and FAQ schema, and align your name, description, and key facts across the properties that feed AI systems and Google's Knowledge Graph.
- Content and authority strategy. We identify the topics worth owning and the citations and mentions that build authority in the sources generative engines already trust.
- Optional file layer. Where a client's audience genuinely includes AI coding agents or developer tools, such as API documentation, we add and automate an llms.txt file as a small, low-cost finishing touch, generated from the same content system as the rest of the site so it cannot go stale.
Every deliverable ships into your own repository and content system, and every recommendation ties back to a source we can point to, not a trend we are chasing.
The Bottom Line
llms.txt is a well-intentioned proposal that solves a real, narrow problem: letting an AI agent orient itself on a site quickly instead of parsing rendered HTML. It is not a ranking factor, Google says so explicitly, and no major AI assistant has confirmed it reads the file at query time. Build one if you run a documentation-heavy site that AI agents actually browse, automate it so it cannot drift out of date, and spend the rest of your GEO budget on the crawlability, structured data, and authority work that every AI system rewards.
Agentixly's GEO team audits AI crawler access, structured data, and content authority together, so you invest in what actually earns citations rather than what is easy to build. Contact us for a technical GEO audit of your site.