Is llms.txt Actually Working? What the Data Says (2026)
Ahrefs, SE Ranking, Otterly, Google and Chrome have all now published something about llms.txt. Read together, they answer the question most guides dodge: nobody is reading it yet, it costs nothing, and the signals that do get you cited are elsewhere.
- Ahrefs looked at 137,000 domains: 97% of the llms.txt files that exist got zero requests in a month, and AI crawlers never asked for the file on sites that did not have one.
- SE Ranking found no relationship between having llms.txt and being cited by AI assistants across roughly 300,000 domains. Removing the file from their model made the model more accurate.
- Google Search says the file will neither help nor harm you, while Chrome's Lighthouse now checks for it. Both are true, and neither is an endorsement.
- Our own audit flags a missing llms.txt, and we explain here exactly how little weight it carries and why we keep the check.
- Publish one if it takes ten minutes. Then spend the real effort on crawler access, firewall blocking, structure, and attribution, which is where the evidence says citations come from.
llms.txt was proposed in September 2024 as a way to hand AI assistants a curated, markdown summary of a site. Two years on it has a spec at llmstxt.org, a generator in every SEO tool including ours, and a checkbox in Chrome's Lighthouse. It also has a growing pile of evidence about whether any AI system reads it, and that evidence is unusually consistent.
This post lays that evidence out, says plainly what our own llms.txt file check does and does not measure, and answers the question guides on this topic tend to dodge: should you bother? The short version is at the end. The long version is why we would rather you read the numbers than take our word for it.
What is llms.txt supposed to do?
The idea is simple. robots.txt tells crawlers where they may go. A sitemap lists every URL. Neither tells a language model what a site is about or which pages matter, and rendered HTML is a noisy thing to feed a model. So llms.txt sits at the site root as a markdown file: an H1 with the site name, a one-line summary, and a short linked list of the pages worth reading. A companion llms-full.txt can carry the full content. The llms.txt guide covers the format and an example, and the llms.txt vs robots.txt piece explains what each file does and does not control.
The premise is that AI assistants would fetch this file the way search engines fetch a sitemap. That premise is what the data tests.
What does the evidence say?
| Source | What was measured | What it found |
|---|---|---|
| Ahrefs, June 2026 | Server logs across 137,000 domains, about 38,000 of them with a valid llms.txt | 97% of the files received zero requests in May 2026. AI bots never requested the file on domains that did not have one. |
| SE Ranking, late 2025 | Roughly 300,000 domains, presence of llms.txt against citation frequency in major AI assistants | About 10% of domains had the file. No correlation with citations, and removing the variable from their prediction model improved its accuracy. |
| Otterly's experiment | 62,100 AI bot visits to their own site over 90 days | 84 requests for llms.txt, or 0.1% of AI crawler traffic. |
| Google Search Central, June 2026 | Official guidance on generative AI features in Google Search | No machine-readable files, AI text files, or Markdown are needed. Google Search does not use llms.txt, and maintaining one "won't harm (nor help)" visibility. |
| Chrome Lighthouse 13.3, May 2026 | New Agentic Browsing audit category | Checks whether the file exists. A missing file is marked not applicable, not failed, because "providing the file is optional at the moment". |
Three different kinds of evidence, one direction. The server logs say the crawlers are not fetching it. The citation study says having it does not change how often you are cited. The platform statements say the biggest search engine ignores it and the biggest browser vendor treats it as optional.
Are AI crawlers requesting the file?
Mostly no, and the detail matters more than the headline. The Ahrefs study is the largest, and its most telling finding is not the 97%. It is that on domains without an llms.txt, AI crawlers never asked for one. A crawler that had been taught to look for the file would probe for it and get a 404, the way Googlebot probes for robots.txt on every site it visits. That probe is absent. The requests that do reach existing files come overwhelmingly from ordinary crawlers and SEO tools, with only around a fifth from anything identifiable as an AI tool.
Otterly's own 90-day log tells the same story at small scale: tens of thousands of AI bot visits, and the file requested 84 times. Google's John Mueller put it bluntly in 2025: none of the AI services have said they are using it, and "you can tell when you look at your server logs that they don't even check for it".
If you want to check your own site, you do not need a study. Grep your access logs for /llms.txt and look at the user agents. On most sites the answer is a handful of SEO checkers, ours included, and nothing that answers questions for real people.
Does having one get you cited more?
No measurable effect, on the largest sample anyone has published. SE Ranking looked at roughly 300,000 domains and how often each was cited across major AI assistants, then tested whether having llms.txt predicted citations. It did not. Their machine-learning model got slightly more accurate when they removed the file as a feature, which is the statistical way of saying it was adding noise, not signal.
This is the finding that should settle the practical question. Even if the crawlers started fetching the file tomorrow, the citation data says the sites that have it are not being cited more than the sites that do not, and the sites that do get cited earned it some other way.
What do Google, OpenAI, and Anthropic actually say?
Google is the only one that has said anything official, and it has said it twice. Its guide to generative AI features in Search, published in May 2026, says you do not need new machine-readable files or Markdown to appear in AI Overviews or AI Mode. A June 2026 update added, in response to community questions, that llms.txt files "won't harm (nor help)" your visibility in Google Search, and that it is fine to keep one for other systems that use it.
At the same time, Chrome's Lighthouse added an Agentic Browsing category in version 13.3 that includes an llms.txt audit. That looks like a contradiction and is not. Lighthouse is measuring readiness for browser agents that act on pages, not ranking in Search, and its own documentation marks a missing file as not applicable rather than a failure. Google Search and Chrome are two products with two audiences, and neither is saying the file earns you a citation.
OpenAI and Anthropic publish llms.txt files for their own developer documentation, and Anthropic asked its docs host to generate one. That tells you the format is useful to a coding agent reading API docs, which it plainly is. It is not a statement that ChatGPT or Claude fetch the file when a consumer asks a question, and neither company has made one.
Why does our audit still check for it?
Because it would be strange to argue for honesty about llms.txt and then hide what our own product does. Here is exactly what the llms.txt file check does. It requests /llms.txt at your site root. If the file is missing, malformed (the first line has to be a markdown # heading), or links to pages that return errors, the check returns a warning. It cannot fail. It is one of nine AI Visibility checks, and the AI Visibility category carries the lowest weight in the score, shared with Quick Wins, so the difference between having the file and not having it is a fraction of a point out of 100. If the request is blocked by a firewall, the check reports that it could not be completed and is excluded from the score entirely, because a blocked request is not evidence the file is absent.
We keep the check for three reasons. It costs the site owner nothing to satisfy. Lighthouse now reports on it, so people will ask. And a file with broken links in it is a real defect: if a crawler ever does follow those links, it lands on a 404. What we do not do is score it as if it were crawler access or schema, because the evidence above says it is not in that league. If a major assistant confirms it reads the file at answer time, we will raise the weight in the same week and say so in the changelog. Until then the check stays where it is.
So should you publish an llms.txt file?
Yes, if it takes ten minutes, and no, if it would take longer than that or displace anything else on this list.
The case for it is cheap insurance. The file is small, harmless, and takes a generator a few seconds to draft from your page titles and descriptions. If the convention is adopted later, you are already there. If it never is, you have lost ten minutes.
The case against spending more than that is everything above. Nobody is reading it, and the citation data says it makes no difference. Every hour spent tuning an llms.txt file is an hour not spent on the signals that demonstrably decide whether an AI assistant can reach, read, and credit your site.
What actually moves AI citations?
The Can ChatGPT see my website? guide frames it as four gates, and llms.txt is not one of them.
- Crawler access in robots.txt. A
Disallowunder OAI-SearchBot, PerplexityBot, or Claude-User removes you from that engine's answers outright. This is the highest-stakes line in the file and the AI crawler access guide lists which agents matter and which are a legitimate licensing choice to block. - Firewall and CDN blocking. robots.txt is a request. A CDN's bot rules can return 403 to the same agents regardless of what robots.txt says, and no robots.txt edit fixes it. This is the silent one, and it is why our checker probes as a browser and as a bot and compares the two.
- Structure the model can parse. Question-format headings, one idea per section, and schema that names the entity behind the page. The how to get cited by AI guide maps each signal to what our checks measure.
- Attribution and freshness. An author with a real name and a
dateModifiedthe page actually honours. Assistants prefer sources they can name and date.
All four are measurable today, all four are free to check on the AI Visibility checker, and the AI Visibility page explains how the nine checks fit together. That is where the ten minutes you did not spend on llms.txt should go.
What would change our mind?
Two things, and we will update this post within the week if either happens. First, a major assistant vendor stating that its answer-time retrieval reads llms.txt, the way Google documents which files Googlebot reads. Second, server-log studies showing the search and citation agents (OAI-SearchBot, PerplexityBot, Claude-SearchBot and their peers) fetching the file at a rate that looks like deliberate behaviour rather than an SEO tool's probe. Either would move llms.txt from insurance to infrastructure, and our check's weight would move with it.
Until then, the honest description of llms.txt in September 2026 is this: an emerging convention, cheap to adopt, unproven for citations, and a poor place to spend effort you could put into the gates that are already decided by the crawlers every day.
FAQ
Does llms.txt help with Google rankings?
No, and this one is not a matter of interpretation. Google's Search Central documentation states that Google Search does not use llms.txt and that having one will neither help nor harm your visibility in Search, including AI Overviews and AI Mode.
Why does Lighthouse check for llms.txt if Google Search ignores it?
Lighthouse's Agentic Browsing category measures how ready a site is for browser agents that act on pages, which is a different audience from Search ranking. Its own documentation marks a missing file as not applicable rather than a failure, because the file is optional.
Is there any harm in publishing one?
Only if it contains dead links or describes pages that no longer exist, since any system that does read it would be sent to a 404. Keep it short, keep it current, and check it with the llms.txt checker after a site restructure. Otherwise the file is harmless.
Should I publish llms-full.txt too?
Only if you already have a clean way to generate it from your content and keep it in sync. The evidence on the summary file applies at least as strongly to the full one, and a stale full-content file is a bigger liability than a stale summary. If you maintain public documentation, it is a reasonable addition for the coding agents that do read docs. For a marketing site, skip it.
Get one AI fix instruction tailored to your platform, free - 3 audits a month, no card. Pro unlocks fixes for every issue.