llms.txt in practice: what the file can do and what we learned from our own
Key points
“We only recommend what we have first tried out on ourselves. With the llms.txt, that mainly showed us how quickly a manually maintained list goes out of date.”
During an audit in August, 81 of 163 addresses were missing from our own llms.txt, including the entire English website. Since September, the file is generated automatically when the website is built.
- The llms.txt is a Markdown file in the root directory that gives AI systems a short description of the website and a list of links. Jeremy Howard proposed it in September 2024.
- Google ignores it for search and its AI features. It does not help there, and it does not hurt.
- An effect on citations in AI answers is not proven. In an analysis by Ahrefs, 97 percent of the existing files were not retrieved by anyone for a month.
- More important for AI visibility is the robots.txt. During our move, a default setting blocked nine AI crawlers.
Chapter 1. What the llms.txt is
The file lives at /llms.txt and begins with a heading, a sentence on positioning and sections with links. Each link carries the title of the page and, where needed, a short note. This is what the beginning of our own file looks like.
It is intended for AI systems and agents that need to grasp a website quickly. The robots.txt, by contrast, regulates access, and the sitemap lists all addresses. The llms.txt replaces neither of them.
Chapter 2. What it does not do
Google writes in its guide to AI features in search that no special files are needed and that search does not use an llms.txt. So it is not a ranking signal.
For the other AI systems, Ahrefs analysed the server logs of 137,210 websites in June 2026. 97 percent of the existing files were retrieved by nobody in May. Of the few retrievals, around one percent came from the systems that assemble answers in AI search, that is, exactly the ones for which the file is usually sold. It is retrieved mainly by developer tools and by audit programmes.
Anyone who sells you an llms.txt as a measure for more AI visibility is selling you a hope. One hour of work is nonetheless worth it, for the reasons in chapter 5.
Chapter 3. What we learned from our own
Our file was maintained by hand at first, and that did not work for long. At the audit on 17 August, the website had 163 addresses. 81 of them were missing from the file. Three German pages were missing as well, and 26 of the 53 article titles were out of date, because we had renamed articles and the file had not been updated. Our AI check went live on 19 August and was not in it for eight days. A page that assesses other websites on whether they tell AI systems what matters was therefore missing from our own file.
Since 23 September, the file is generated when the website is built. The titles come word for word from the articles, and the build aborts if a page is missing from the sitemap. On the same day we drew a second lesson. The file contained our build log as a comment, readable for anyone. Since then, nothing goes into it that is not intended for readers.
Chapter 4. The bigger lever lies in the robots.txt
More important for us than the llms.txt was a file we had hardly thought of during the move. Cloudflare places its own block in front of the robots.txt it serves, and that block appears nowhere in the project. On go-live day, it blocked nine crawlers, including GPTBot, ClaudeBot and Google-Extended.
At the same time, the bot settings of the same zone were set to “allow”, and within 24 hours, 17 of 17 crawler requests were let through. A crawler that adheres to the robots.txt does not even come in such a case. Most of the blocked bots collect training data. According to Google, Google-Extended uses a single signal to control the training of Gemini and the use of content in its answers, and that decision should be made deliberately.
Since then, at every go-live, we read the robots.txt as delivered and the CDN’s bot settings in the same pass. Whether your website lets in the common AI systems is shown by the free AI check in ten tests.
Chapter 5. How to create an llms.txt in five steps
- Positioning in one sentence. It stands at the very top and carries the whole file.
- Select pages. Service pages, industries, contact and the most important articles. A file with 500 lines is a second sitemap.
- Notes only where the title is not enough. “AI visibility” explains itself, a tool such as our AI check does not.
- Automate the generation. From the sitemap or CMS, when the website is built. Otherwise the file goes out of date with the next article, as ours did.
- Treat it like code. No instructions to models, no superlatives, only links to your own content.
The file is most likely worth it if your customers work with AI tools for developers, or if agents are supposed to carry out tasks on your website.
llms.txt toolkit: your file in five fields
Everything stays in your browser. Nothing is stored or sent.
Your llms.txt
# Sample Machinery Ltd > Sample Machinery develops and manufactures special machines for the packaging industry in the DACH region. ## Services - [Special machines](https://www.muster-maschinenbau.de/sondermaschinen/) ## Optional
Frequently asked questions
Do ChatGPT, Copilot, Gemini and Claude read the llms.txt?
Rarely. In the Ahrefs analysis, only around one percent of the retrievals came from the AI search systems. Whether your content appears in answers is decided by the robots.txt and by the content itself.
Does the llms.txt have to be in the sitemap or in the robots.txt?
No. It lies in the root directory and is found via its fixed address.
Why does LILOX have an llms.txt if it brings little?
Because it costs almost nothing and is part of a basic technical setup that we also recommend to clients. And because our own file showed us how quickly a manually maintained list goes out of date.
Revised in October 2026: rewritten with the experience from our own llms.txt, considerably shortened, author added.
Sources
- Jeremy Howard / Answer.AI, “The /llms.txt file”: https://llmstxt.org/
- Google Search Central, “Optimizing your website for generative AI features on Google Search”: https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
- Ahrefs, “We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read”, 15 June 2026: https://ahrefs.com/blog/llmstxt-study/
- Google, “List of Google’s common crawlers” (Google-Extended): https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- Own data: llms.txt and robots.txt of lilox.io, project logs from August to September 2026.
