- Ahrefs analysed 137,000 domains and found 97% of llms.txt files received zero requests in May 2026 — no bots, no humans
- A separate SE Ranking study of 300,000 domains found no statistical correlation between having llms.txt and AI citation frequency
- Google's John Mueller has publicly described llms.txt as "not done for search" — at most a temporary aid for coding agents, not a visibility lever
- The genuine tension is between Google's own guidance (you don't need it) and Chrome's Lighthouse tool (which audits for it) — these answer different questions, not contradictory ones
- The fundamentals — content quality, technical SEO, internal linking, and structured data — remain the actual drivers of AI visibility, with or without llms.txt
If you have spent any time in SEO or marketing circles over the past few weeks, you have probably seen at least one hot take on llms.txt. Some say Google quietly reversed its position. Some say Chrome just made it mandatory. Some say the data proves it is dead on arrival. A few are still cheerfully telling clients to "ship it Monday" because it is "the robots.txt of the AI era."
We went through the actual sources — Google's own documentation, Chrome's developer docs, the recorded comments from Google's Search team, and two of the largest independent studies done on this file so far — to figure out what is actually true, what is speculation, and what businesses should genuinely do about it.
Short version: the file is real, it exists, and you can publish one in minutes. Whether it does anything for your AI visibility is a separate question entirely, and the evidence so far says: not really, not yet.
What llms.txt Actually Is (And What It Isn't)
Llms.txt is a single markdown index file, hosted at a site's root, originally proposed in 2024 by Jeremy Howard, co-founder of Answer.AI and fast.ai. The idea was simple: give large language models a clean, curated summary of a site's most important pages so an AI system doesn't need to crawl the entire site to understand what it is.
That is the entire original scope. It was designed as a navigation aid for AI agents and coding tools — not, originally, as an SEO or AI-visibility tactic. ("AI visibility" simply means how often a brand or website gets mentioned, linked, or quoted inside an AI-generated answer — the AI-search equivalent of ranking on Google.) The "AI visibility" framing came later, layered on by parts of the SEO industry as adoption spread, largely on the speculation that AI platforms would eventually reward sites that had one.
This distinction matters because most of the confusion in the market right now comes from people treating llms.txt as if it behaves like robots.txt or sitemap.xml — files that search engines are built to actively look for and act on. Llms.txt has no such enforced relationship with any major AI platform.
What an llms.txt File Actually Looks Like
Before getting into the data, it helps to see the file itself, since a lot of the confusion online comes from people who have never actually opened one. Structurally, llms.txt is closer to a short README than anything resembling SEO markup.
The format is plain markdown, typically following a loose convention:
- An H1 with the site or company name
- A one or two-line summary describing what the site or product does
- One or more H2 sections grouping links by category — documentation, product pages, blog, pricing
- A flat list of markdown links under each section, each with a short description
If you do want to create one, the practical version is genuinely simple: list your most important pages, write a one-line description for each, and host the file at yourdomain.com/llms.txt. Most modern CMS platforms and SEO plugins can generate a basic version automatically.
The Data: What Happens When You Actually Check the Logs
Opinions on llms.txt are everywhere. Server log data is rarer. Two independent studies this year have actually looked at what happens on real websites, and both point in the same direction.
Ahrefs: 137,000 Domains, 97% Silence
In June 2026, Ahrefs published the largest traffic study of llms.txt to date, analysing server logs and live traffic across 137,000 domains in its Web Analytics dataset.
Of the roughly 38,000 domains that had a valid file, only about 1,100 ever received a single request to it. Of those requests, 96% came from bots — and most of those bots were not AI tools at all. They were SEO audit tools, unidentified crawlers, and tech-profiling tools like BuiltWith. Ahrefs also found that named AI tools made up only 19.5% of the small slice of traffic that did occur, with GPTBot and Claude-Code leading — and notably, both of those are coding-related bots, not the live AI search assistants most businesses are trying to be visible to.
SE Ranking: 300,000 Domains, Zero Correlation
Earlier, SE Ranking ran a larger-scale study across roughly 300,000 domains, this time testing specifically whether having an llms.txt file correlated with how often a domain gets cited in AI-generated answers. ("Cited" here just means: when you ask an AI tool a question, does it mention or link to that website as a source.)
They used both statistical correlation testing and a machine-learning model trained to predict citations (the kind of tool that looks for patterns across thousands of data points at once). The result: no statistically significant relationship — in plain terms, having the file made no detectable difference either way. In fact, when they removed llms.txt as a factor the model was tracking, the model's predictions actually got more accurate — meaning the file was adding confusion to the data, not useful signal.
| Study | Sample Size | Key Finding |
|---|---|---|
| Ahrefs (Jun 2026) | 137,000 domains | 97% of llms.txt files received zero requests in a month |
| SE Ranking (Nov 2025) | 300,000 domains | No statistical correlation with AI citation frequency |
Two independent datasets, two different methodologies, the same conclusion. That is not proof the file will never matter — but it is strong evidence that, as of mid-2026, it is not moving the needle on AI visibility.
What Google Actually Said (Because It Is Not What Most Posts Claim)
This is where the floating LinkedIn posts get it wrong. The original claim — that "Google clarified its guidance only applies to Google Search, and gave the green light for llms.txt on other platforms" — is not an accurate read of what Google published.
Google's own guide, Optimizing your website for generative AI features on Google Search, addresses llms.txt directly in a section literally titled "Mythbusting generative AI search: what you don't need to do." The guidance states plainly that you don't need to create "new machine readable files, AI text files, markup, or Markdown" to appear in Google Search, including its AI features, because Google Search itself doesn't use them.
Crucially, Google does add one line that has been seized on and overstated: it says it's "completely fine" to maintain such files for other services or systems that use them — but immediately follows that with: doing so "won't harm (nor help) your visibility or rankings in Google Search, as Google Search ignores them." That is a neutral non-endorsement, not a green light or a strategic recommendation. Google is not telling you llms.txt helps elsewhere. It is telling you Google itself doesn't care either way.
Source: John Mueller, Google Senior Search Analyst, on the Search Off the Record podcast, responding to a question pressed by SEO consultant Lily Ray
Mueller went further, describing llms.txt as "not done for search" and, at most, a "temporary crutch, perhaps to save some tokens" for AI coding tools trying to parse developer documentation — not something the average business website needs to worry about. He also pointed out that site owners checking their own server logs will typically find very little AI agent traffic to begin with, a claim the Ahrefs data above independently backs up.
So Why Does Chrome's Lighthouse Tool Check For It?
This is the genuine tension, and it is worth taking seriously rather than dismissing as "Google contradicting itself." It helps to remember that Google Search (the search engine that answers your questions) and Google Chrome (the web browser app) are different products built by different teams inside the same company — so it's entirely possible for one team to say "you don't need this" while another team builds a tool that checks for it, without that being a contradiction.
Days after publishing the guidance above, Chrome's developer team shipped an llms.txt check inside Lighthouse's experimental Agentic Browsing audits. (Lighthouse is a free Chrome tool that scans websites and grades them on things like speed and accessibility — "agentic browsing" just means how easily an AI assistant can navigate and use a site on a person's behalf, similar to how a person might ask an AI to "book me a flight" and have it actually click through a website to do it.) The documentation frames it differently from a search-visibility lever: without the file, the reasoning goes, an AI browsing agent that has already landed on your site may need to spend more time crawling its structure to understand it.
These are two different questions wearing the same three letters. "Does Google Search need this file to find and rank you" and "does a browsing agent that has already arrived at your site navigate it more efficiently with this file" are not the same question, and Ahrefs found exactly this in the data: the Lighthouse audit itself produced roughly 1 in 1,000 of all the fetches they recorded. It is a real signal, but a faint one, aimed at a narrow use case — not a sign that AI discovery broadly depends on the file.
Our View
Having gone through the primary sources rather than the secondhand takes, here's where we land:
This also mirrors a wider pattern we keep seeing in the broader AI search conversation right now: a self-declared file is, by design, the easiest kind of signal for an AI system to be skeptical of. As one analysis put it, AI citation systems are trained to synthesize information from across the web — publication authority, third-party corroboration, editorial credibility — and a file you write about yourself sits at the bottom of that hierarchy, not the top.
What the Market Is Saying
It's worth noting the discussion hasn't settled even among practitioners who have looked closely at the data. Some in the SEO community have pointed out that Chrome auditing for the file at all, even as a minor agentic-browsing check, suggests the file could become a more meaningful layer for AI discoverability over time, rather than dismissing it outright. Others maintain it's currently indistinguishable from a low-priority "nice-to-have" technical file. Both positions are reasonable readings of an evolving, still-thin evidence base — which is exactly why we'd caution against anyone declaring the debate closed in either direction.
What to Actually Prioritise Instead
Every study cited above arrives at roughly the same closing recommendation, and it's not a new one: the fundamentals still decide AI visibility, with or without llms.txt.
- ✓ Publish original, experience-backed content rather than commodity summaries
- ✓ Maintain clean technical SEO — crawlability, indexability, fast pages (the basic housekeeping that makes a website easy for any search engine, AI or otherwise, to read properly)
- ✓ Build genuine topical authority through connected content, not isolated articles
- ✓ Use structured data and schema markup where relevant (extra labels added to a webpage's code that tell search engines exactly what each piece of content is — a price, a review, a business address — rather than making them guess)
- ✓ Earn third-party validation — mentions, citations, and authority signals beyond your own site
If you want a deeper breakdown of how those fundamentals translate specifically into AI search visibility, our guides on optimising your website for AI search and measuring AI SEO performance go into the parts of this that the data actually supports.
Final Thoughts
Llms.txt is not a scam, and it is not robots.txt. It's a small, genuinely low-effort file with a narrow, mostly developer-facing purpose that got swept into the AI SEO hype cycle faster than anyone had data to support it. The two largest independent studies done on it so far — 137,000 domains and 300,000 domains respectively — both arrive at the same place: no detectable AI visibility benefit today.
That could change. Google itself leaves the door open, and Chrome's Lighthouse audit shows at least one part of the ecosystem is paying attention to it for agentic use cases. But "might matter eventually" is a different claim from "you need this now," and right now, the second claim isn't supported by the evidence. Spend the bulk of your effort on the fundamentals. Add the file if it costs you nothing to do so. Don't let anyone sell it to you as the new robots.txt.
