llms.txt: Why Your Site Needs It and What to Put Inside

Learn what llms.txt is, how it helps AI tools find your content, what to put in the file, and how it differs from robots.txt and sitemaps.

ADS Beast editorial teamPublished Updated 13 min read

This article was prepared with AI. The ADS Beast editorial team is responsible for its content.

An llms.txt file is a plain text document at the root of your domain that gives AI models a structured summary of your website's most important content. It is meant to help AI tools such as ChatGPT and Perplexity read your site without parsing JavaScript-heavy pages, although no provider is obliged to use it. This article explains what to put inside, where to place it, and what it can and cannot do for your visibility in AI-generated answers.

Short version

  • llms.txt is a markdown-style text file placed at the root of your domain, like https://example.com/llms.txt.
  • It tells AI models which pages matter most and what your site is about, in plain language.
  • Keep it short: a few dozen links to core pages at most, each with a short description. The specification sets no limit.
  • It complements robots.txt and sitemaps; it does not replace them.
  • AI search engines and chatbots now read websites directly, and this file offers them a clean path if they choose to use it.

What is an llms.txt file and where does it go?

An llms.txt file is a plain text document that provides a structured, machine-readable summary of a website's content for AI models. You place it at the root of your domain, so the full URL looks like https://example.com/llms.txt. The file uses simple markdown-style formatting: a title, a summary block, and a list of links with short descriptions.

The format was proposed in September 2024 by Jeremy Howard (llmstxt.org) as a way to help large language models understand websites faster and more accurately. Think of it as an elevator pitch for AI crawlers. Instead of scanning every page and guessing what matters, the model reads your llms.txt and knows immediately what your site covers and which pages carry the most weight.

The file itself is lightweight. It has no code, no scripts, and no tracking pixels. Any text editor can create one, and you can host it on any static file server. The simplicity is the point.

Create your llms.txt file. Write your summary: One sentence stating who you are and what you offer.; Pick your top pages: Choose the pages that answer commercial questions or describe core services.; Write clear descriptions: Each link gets a short description of what the page contains.; Upload to d
Five steps from empty file to live llms.txt on your domain root.

Why does your website need an llms.txt file?

Because AI search engines and chatbots now read websites directly, and they often struggle with JavaScript-heavy pages. When a model cannot render your site, it either skips you entirely or guesses based on fragments of text. An llms.txt file removes that guesswork.

The shift is real. People now ask ChatGPT for product recommendations, Perplexity for research summaries, and Google's AI Overviews for quick answers. These tools cite sources, and they prefer pages they can read reliably. If your site is built on React, Angular, or another framework that renders content client-side, an AI crawler may see an empty shell.

Your llms.txt file is meant to solve that problem. It gives AI models a clean, text-based path to your key pages. This can help your visibility in AI-generated answers, but adoption is up to each provider: Google, for one, says you do not need new machine-readable or AI text files to appear in its AI features (Google Search Central).

The file also helps with accuracy. When an AI model reads your entire site, it can mix up services, prices, or contact details. A well-structured llms.txt file tells the model exactly what you offer and which pages confirm it. The aim is fewer mix-ups and more accurate citations.

What should you put inside the llms.txt file?

Start with an H1 carrying your site name and a one-sentence summary, then add lists of links to your most important pages, each with a short description. Keep the file short, and leave out boilerplate pages like privacy policies or login screens.

# Example Company

> Example Company provides accounting software for small businesses in the UK.

- [Pricing](https://example.com/pricing): Plans for freelancers, startups, and growing teams
- [Features](https://example.com/features): Invoicing, expense tracking, and VAT reports
- [Integrations](https://example.com/integrations): Works with Stripe, PayPal, and Shopify
- [Help Center](https://example.com/help): Setup guides and troubleshooting articles

In a full file the links sit under H2 headings such as ## Docs or ## Products, and a section named ## Optional marks links a model can skip when it needs a shorter context. The summary line matters most. It should be a single sentence that states who you are and what you do. Avoid marketing language. Write it the way you would explain your business to a colleague, not the way you would write an ad.

Each link should have a description of no more than 10 to 15 words. The description tells the AI model what it will find on that page. If the page answers a specific question, say so. For example, "Setup guide for connecting your bank account" is better than "Documentation".

Leave out pages that do not help a user make a decision: privacy policies, terms of service, login screens, and thank-you pages. These add noise and dilute the signal.

Which pages deserve a spot in the file?

Your homepage always goes first. After that, prioritize pages that answer commercial questions: pricing, features, comparisons, case studies, and product documentation. If you run a blog, include your most authoritative articles, not every post you have ever published.

Here is a quick filter for deciding what belongs in the file:

  1. Does this page help someone decide whether to buy, sign up, or contact you?
  2. Does this page answer a question people frequently ask AI assistants?
  3. Does this page represent a core service or product line?

If you answered yes to at least two of these, the page belongs in the file. If not, leave it out.

llms.txt vs robots.txt vs sitemap. robots.txt: Tells crawlers what not to scan. Read by all crawlers. Usually short.; sitemap.xml: Lists every URL for indexing. Read by search engines. Can hold thousands of URLs.; llms.txt: Summarizes site purpose and top pages for AI. Read by AI tools that support
Each file serves a different crawler with a different purpose.

How is llms.txt different from robots.txt or a sitemap?

Robots.txt tells crawlers what not to scan, and sitemaps list every URL for search engines. The llms.txt file goes further by telling AI models what your content is about and which links matter most, so they don't have to parse your whole site.

FilePurposeWho reads itTypical size
robots.txtBlocks or allows crawling of specific pathsAll crawlers, including Googlebot and BingbotUsually short
sitemap.xmlLists every URL for indexingSearch engines primarilyCan contain thousands of URLs
llms.txtSummarizes site purpose and top pages for AI modelsAI assistants and AI search toolsShort by design

The three files work together. Robots.txt sets the boundaries. The sitemap ensures search engines know every page exists. The llms.txt file tells AI models which of those pages actually matter and why.

A common mistake is treating llms.txt as a replacement for a sitemap. It is not. Search engines still need your sitemap to discover new pages. And you still need robots.txt to block sensitive areas. Add llms.txt alongside them, not instead of them.

Another difference is the format. Sitemaps use XML, which is verbose and rigid. Robots.txt uses plain text but only supports directives like allow and disallow. The llms.txt file uses markdown-style formatting that carries semantic meaning: a title, a summary, and annotated links.

How many links should you include in the llms.txt file?

There is no official limit. Aim for a short list that covers your core services or articles: a few dozen links at most. Adding too many dilutes the signal.

The upper limit matters. An AI model reading your llms.txt file treats it as a curated list. If you include hundreds of links, nothing in the file tells the model which pages are truly important. The signal becomes noise.

For most small and medium businesses, fifteen to twenty links is a sensible start. You cover your main service pages, your pricing page, your most popular articles, and your contact information. If you run an e-commerce store, include category pages rather than individual product pages. If you run a SaaS company, include feature pages and documentation.

For enterprise sites with hundreds of products or services, consider creating multiple llms.txt files for different sections. The main file points to section hubs, and each hub has its own llms.txt file. This keeps each file short while still covering your full catalog.

The quality of your descriptions matters more than the number of links. Ten links with precise descriptions are more useful than thirty links with vague ones. Write each description as if it is the only thing the AI model will read about that page.

How does llms.txt fit into your AI visibility strategy?

An llms.txt file is one part of a broader approach to AI search optimization. You still need clear page titles, descriptive headings, and content that answers real questions. The file does not fix a poorly structured site. It simply makes your best content easier for AI models to find.

You should also monitor what AI assistants actually say about your company. The answers they give shape what potential customers believe about you. If an AI model describes your product incorrectly, no llms.txt file will fix that. You need to audit the answers and adjust both your content and your file. A practical guide for that process is the What AI Answers Say About Your Company: Audit Guide.

The file also works alongside your tracking setup. If you want to measure how much traffic comes from AI referrals, you need reliable conversion tracking. Browser-based tracking can lose part of the data, so many teams add server-side tracking. The article on server-side conversion tracking: how to enable without double-counting explains how to set that up without inflating your numbers.

And when you review performance, pay attention to attribution windows. A visit that starts in an AI assistant may convert days later. The piece on the Attribution Window: Why the Same Period Shows Different Numbers shows why your reports may disagree depending on which window you look at.

Common mistakes when creating an llms.txt file

The most frequent error is treating the file like a sitemap and dumping every URL into it. That defeats the purpose. The file is a curated summary, not an index.

Another mistake is writing vague descriptions. A link that says "Products" tells the AI model nothing. A link that says "Cloud storage plans with 2TB to 10TB options" tells it exactly what the page offers. Write for clarity, not for length.

Some site owners forget to update the file. Your site changes, new products launch, old articles get removed. The llms.txt file needs the same maintenance as your sitemap. Set a reminder to review it quarterly.

A subtle issue is the summary line. Keep the summary to a sentence or two; a long paragraph defeats the purpose of a quick overview. And if you operate in multiple languages, consider creating separate llms.txt files for each language version of your site.

Finally, do not block the file in robots.txt. A disallow rule that covers it blocks the very crawlers you want to reach. The file is public by design, just like your homepage.

Your next step: create the file today

A basic file is quick to create, and you can host it as a static file on any server. Write a one-sentence summary of your business, list your top 15 to 20 pages with clear descriptions, and upload the file to your domain root. Then verify it loads by visiting https://yourdomain.com/llms.txt in a browser.

After that, integrate llms.txt into how you measure and improve your AI presence. If you manage marketing for multiple clients or run paid campaigns, you need to know which traffic actually converts. For the paid side, ADS Beast is a self-service ad platform: campaigns are created paused, and launch and budget changes wait for your approval.

See how this works in ADS Beast.