When an AI assistant lands on your website, what should it read first? That is the question llms.txt answers. The file sits at the root of your domain and lists your most important pages in plain Markdown. Search engines have robots.txt and XML sitemaps. Large language models such as ChatGPT, Claude, and Gemini now have an entry point of their own. At RankWit we treat it as one of the first steps of any GEO project.
llms.txt is a Markdown-formatted text file, placed at the root of a website, that tells large language models which content matters most and how to interpret it.
The format is deliberately simple. No backend work, no new markup language. You write a short file, upload it once, and every AI crawler that looks for it knows where your best content lives.
Where does llms.txt come from?
The proposal was published in September 2024 by Jeremy Howard, co-founder of Answer.AI and a well-known AI researcher. His argument is direct: websites should have a say in how AI systems read and reuse their content, and a file convention is the simplest way to give them one.
The community site llmstxt.org maintains the specification, collects examples, and monitors adoption. Documentation platforms were among the first movers, and many developer tools now publish the file by default.
What problem does it solve?
Websites are built for people, and it shows. A typical page carries navigation menus, cookie banners, tracking scripts, ads, and layout code. The actual content is a small fraction of the HTML.
Language models pay a price for that noise. They work with limited context windows, so every token spent parsing a menu is a token not spent reading your product documentation. When an assistant reconstructs what your company does from cluttered markup, mistakes creep in.
llms.txt removes the guesswork. It hands the model a curated map: here are the docs, here is the FAQ, here is the pricing page, and this is what each one covers. When ChatGPT or Perplexity talks about your product, the odds improve that it gets the story right and points readers to the source.
How is llms.txt different from robots.txt?
The two files look alike and live in the same place, but they answer different questions.
- robots.txt handles exclusion. It tells crawlers which paths they may or may not access, and it can block AI bots such as GPTBot entirely.
- llms.txt handles curation. It tells language models which content deserves attention and provides context for each link.
- sitemap.xml handles inventory. It lists every indexable URL without saying which ones matter.
They complement each other. A site can block training crawlers in robots.txt and still use llms.txt to guide the assistants it welcomes.
What goes inside the file?
The specification expects a predictable structure, which is exactly what makes it machine-friendly:
- An H1 heading with the name of the site or project.
- A short blockquote that summarizes what the site offers.
- Sections of links, each with a one-line description of the target page.
- An optional llms-full.txt with expanded content for models that have large context windows.
The whole file reads like a well-organized index page. A human can check it at a glance, and a model can parse it without touching a single line of layout code.
How do you create an llms.txt file?
- Select the pages that explain your business best: documentation, FAQs, guides, pricing.
- Write one line of context for each link. Say what the page contains, not how great it is.
- Assemble the file in Markdown, starting with the H1 and the blockquote summary.
- Upload it to your site root so it resolves at yoursite.com/llms.txt.
- Review it whenever you publish or retire important content. A stale map misleads.
For most sites this is an afternoon of work, and the first three steps double as a useful content audit.
Will AI engines actually read it?
Here is the honest part. Adoption among the major AI providers is uneven, and none of them has committed to fetching the file on a fixed schedule. Some tools check it, others ignore it.
We think the balance still favors publishing one. The cost is close to zero, search engines are unaffected, and your server logs will show which AI crawlers request the file. GPTBot, ClaudeBot, and PerplexityBot all identify themselves in the user agent string, so the experiment measures itself.
There is also a positioning argument. Generative engines reward sites that are easy to read and easy to quote. The habit of maintaining a curated content map pushes you in that direction, llms.txt or not. Our guide on how ChatGPT chooses its sources shows what these systems look for once they reach your pages.
When should you add one?
Now, if your site contains content people rely on:
- Documentation or API references
- Tutorials and how-to guides
- Product and service pages
- Knowledge bases and FAQs
- Support and contact information
The rule of thumb holds: if it is valuable to users, it is valuable to LLMs.
The file is a starting point, not a strategy. Discovery is moving from ranked links to generated answers, a shift we cover in our article on the move from search engines to generative engines. Publishing llms.txt makes you easier to read. Knowing whether ChatGPT, Perplexity, and Gemini actually cite you is the other half, and that is what RankWit are built for.
FAQ
Is llms.txt an official web standard?
No. It is a community proposal, published by Jeremy Howard in September 2024 and documented at llmstxt.org. No standards body has ratified it, and AI companies support it voluntarily. Adoption has spread anyway, especially among documentation-heavy sites, because the format is simple and the downside is close to zero.
Does llms.txt replace robots.txt or my sitemap?
No. The three files do different jobs. robots.txt controls which crawlers can access which paths, sitemap.xml lists your indexable URLs for search engines, and llms.txt curates the content you want language models to prioritize. Keep all three, and make sure they do not contradict each other.
Can llms.txt hurt my SEO?
No. Search engines do not use the file for ranking, so publishing it cannot cost you positions. The only real risk is neglect: if the file points to outdated or deleted pages, models may work from stale information. Treat it like any other public page and keep it current.
How can I tell if AI bots are reading my llms.txt?
Check your server logs for requests to /llms.txt and filter by user agent. OpenAI's GPTBot, Anthropic's ClaudeBot, and PerplexityBot all declare themselves. If you see regular fetches, the file is being consumed. If not, you have lost nothing: the file still works as documentation of your priorities.