AI search visibility
llms.txt: do you actually need one?
In short. For most websites, no. llms.txt is a proposed Markdown file that lists your best content for large language models, but crawler-log studies show the AI engines almost never fetch it, and Google has confirmed it ignores the file entirely. It is worth adding only in narrow cases, mainly documentation-heavy software products whose users point coding assistants at their docs. For everyone else, publishing one is low effort and low reward, and it will not get you cited.
llms.txt is one of the most talked-about ideas in AI search, and one of the most misunderstood. The file was proposed by Jeremy Howard of Answer.AI in September 2024 as a way to hand large language models a clean, curated map of a site instead of forcing them to parse cluttered HTML. The pitch is appealing: give the AI engines a menu of your best pages and they will understand you better. The evidence, unfortunately, tells a more sober story. This guide separates the proposal from the reality so you can decide whether the file deserves an hour of your time.
What is llms.txt?
llms.txt is a plain Markdown file placed at the root of a domain (yoursite.com/llms.txt) that lists your most important pages so language models can find them quickly. It is not a blocking file and it is not robots.txt: it does not control crawling or indexing. Think of it as a curated table of contents written for machines rather than a rulebook.
The format is deliberately simple. It uses an H1 with your brand name, an optional one-line summary in a blockquote, H2 headings to group sections, and bullet links in the form - [Title](URL): short description. A companion file, llms-full.txt, bundles your entire documentation into one long text file for models that want the raw content. The full specification lives at llmstxt.org and is still an informal community proposal, not an adopted web standard.
Do the AI engines actually read llms.txt?
Overwhelmingly, no. When researchers check real server logs, they find that the crawlers behind ChatGPT, Claude, Perplexity, and Google request the HTML of your pages and skip the llms.txt file almost entirely. The intended readers are simply not reading it.
An analysis of server logs from 137,000 domains found that 97% of llms.txt files received zero requests in May 2026, as reported by PPC Land. The same reporting shows adoption is climbing fast (llms.txt instances grew 8.8x in twelve months, from 4,088 to 36,120 sites per an Originality.ai scan of over 3 million domains), yet the files sit untouched. Growing supply with almost no demand is the core problem: people keep publishing the file because guides tell them to, while the AI engines keep ignoring it.
Does Google use llms.txt?
No. Google has stated plainly that its Search products, including AI Overviews and AI Mode, do not use llms.txt, and that you do not need to create machine-readable AI files to appear in them. Google's John Mueller has compared the idea to the discredited keywords meta tag, and noted that "you can tell when you look at your server logs that they don't even check for it."
The reasoning matters more than the ruling. A signal the site owner fully controls is easy to game, so search engines discount it, exactly the fate the keywords meta tag met in the 1990s. Google guidance can vary by product, as Search Engine Journal has documented, but for organic Search and its AI features the answer is settled: llms.txt is not an input.
Does ChatGPT use llms.txt?
OpenAI has never published anything saying that ChatGPT, GPTBot, or its OAI-SearchBot crawler parse llms.txt or treat it as a special input. Site owners occasionally see a bot request /llms.txt in their logs, but a fetch only proves the file is reachable. It does not prove the content was parsed, prioritized, or used to shape an answer.
Claude and Perplexity are the two engines most often cited as retrieving the file today, and it does have real traction inside developer tooling: coding assistants, agent frameworks, and retrieval pipelines that point at documentation. That is a meaningful niche, but it is not the same as "the AI engines read your llms.txt and cite you more." For consumer AI search, no major provider has committed to honoring it.
What does an llms.txt file look like?
A minimal file is short and readable. Here is the shape of one, described rather than rendered: a top heading with your brand, a one-sentence summary, then grouped links to your key pages with a brief note on each. That is the entire spec in practice.
The table below shows how llms.txt differs from the two files people confuse it with, so you know which job each one does.
| File | Purpose | Honored by major engines? |
|---|---|---|
| robots.txt | Allow or block crawlers by path | Yes, long-standing standard |
| sitemap.xml | List all indexable URLs for discovery | Yes, supported by search engines |
| llms.txt | Curate best pages for language models | No, largely ignored today |
Because it is so easy to write, the temptation is to treat creation as progress. Publishing the file is the cheap part. Getting anything back from it is where the proposal falls short.
Who should actually create an llms.txt file?
A small set of sites. If your users regularly feed your documentation into coding assistants or agents, llms.txt is a genuine convenience because those tools do read it. If you are a marketing site, a local business, a blog, or a store hoping for more AI citations, it will do nothing measurable. This use-case verdict is the piece most guides skip: they explain how to make the file without telling you whether your site is one of the few that benefits.
| Your site | Should you bother? | Why |
|---|---|---|
| Docs-heavy SaaS or dev tool | Yes | Coding agents and RAG pipelines actually consume it |
| Marketing or lead-gen site | Skip | Consumer AI engines ignore it; no citation lift |
| Ecommerce store | Skip | Product data comes from feeds and structured markup, not this file |
| Local business | Skip | Local AI answers pull from reviews and business listings |
| Blog or publisher | Optional, low priority | Harmless, but your HTML and structure do the work |
Does llms.txt help you get cited by AI?
There is no evidence that it does. A controlled experiment that tracked 10 sites after adding llms.txt found that only two saw any AI traffic increase, and in both cases the file was not the cause: the gains traced back to structured content, technical fixes, and external validation. One site in the study actually declined 19.7% over the window. Correlation studies looking at AI citations have found no relationship with the presence of an llms.txt file.
The reason is straightforward. The AI engines cite passages they can lift cleanly from your live HTML, and they weigh how often your brand is mentioned across the wider web. A file most of them never open cannot influence either. If you want citations, the work happens on the page and off the page, not in a root-level text file.
What should you do instead to show up in AI answers?
Put the effort where the AI engines actually look. That means writing answer-first passages, self-contained sections, clear definitions, comparison tables, and sourced statistics, so a model can extract a clean, quotable chunk from your live page. Our guide to structuring content for AI citations walks through the passage-level formatting that correlates with being quoted, and the generative engine optimization primer covers the on-page and off-page signals that move the needle. If you are chasing a specific engine, our breakdown of how to get cited by ChatGPT gets more specific.
Measurement matters too, since AI visibility does not show up in classic rank trackers. Tools that log whether each engine mentions, cites, or recommends you across a panel of buyer prompts tell you far more than an llms.txt file ever will. Sorank is one such platform (it tracks AI visibility alongside content production, from $99/mo with 3 days free). Disclosure: this site is operated by Stone Rank, the team behind Sorank, and we cover competing tools on the same terms in our buyers guide. Whichever tool you use, tracking beats guessing.
Conclusion
For the vast majority of sites, you do not need an llms.txt file. The proposal is elegant, but crawler logs show the AI engines rarely fetch it, Google has confirmed it ignores it, and controlled tests show no citation lift. The honest exception is documentation-heavy software whose users point coding assistants at the docs. Everyone else should spend that hour making live pages easy to quote and tracking how the AI engines actually treat their brand. See how the leading platforms compare in our ranking of the best AI SEO software.
Compare the best AI SEO software
Frequently asked questions
Do I need an llms.txt file?
Most sites do not. Adding one is low effort but crawler-log studies show the AI engines almost never read it, so it will not improve your citations. The clear exception is documentation-heavy software products whose users feed the docs into coding assistants, which do consume the file.
Does ChatGPT use llms.txt?
OpenAI has not confirmed that ChatGPT or its crawlers parse llms.txt. You may occasionally see a bot request the file in your logs, but a fetch does not prove the content was used to shape an answer. Treat it as unsupported for consumer AI search.
Does Google use llms.txt?
No. Google has stated its Search products, including AI Overviews and AI Mode, do not use llms.txt, and that you do not need machine-readable AI files to appear in them. John Mueller has compared the idea to the long-discredited keywords meta tag.