llms.txt: How to Feed Your Content to AI Search Engines — Step-by-Step Guide for Beginners

Introduction: Why Your Website Needs llms.txt

Just a couple of years ago, the user's path to your content looked simple: a search query, a list of links, a click. In 2026, the picture is different. More and more people are asking questions to AI assistants and AI search engines, and those tools read websites, summarize them, and cite them. If a model doesn't understand what your site is about, it will either ignore it or misrepresent it. The llms.txt file solves exactly this problem: it explains to language models what you have and where the most important content lives.

This practical guide will take you from a blank notepad to a working llms.txt file on your domain. No theory for theory's sake — just concrete actions with verification at every step.

What You'll End Up With

• A ready-to-go llms.txt file sitting in your site's root and accessible at your-domain/llms.txt.
• An extended llms-full.txt version with the full text of key pages.
• An understanding of which pages are worth showing to AI and which aren't.
• A verification setup in place: you'll know the file is readable and that AI bots are reaching it.
• A foundation for further work with generative optimization — that is, your brand's visibility in AI answers.

Who This Guide Is For

For business owners, marketers, media buyers, and developers who want their product, landing page, or blog to show up correctly in AI assistant answers. If you have a website and you know how to upload files to it — or know who does — you have enough skills. There's a dedicated section at the end for those who want to automate the process and monitor AI bot behavior through logs.

What You Need to Know in Advance

• How your site is built: on a website builder, a CMS like WordPress, or custom code.
• Where your site files are stored and how to access them: hosting panel, FTP, repository.
• A basic understanding of Markdown: headings with hash marks, lists with dashes, links in square and round brackets. If you're not familiar, no worries — we'll explain with examples.

How Much Time It Takes

A minimal version for a site with up to 30 pages takes about an hour. The full version with llms-full.txt, verification, and server header setup will take two to three hours. For large catalogs and documentation, set aside a full day — but the result will be noticeably better quality.

Preparation

Good preparation saves more time than any hack. Go through the list below and make sure everything is at hand.

Essential Tools and Access

1. Access to your site files. This could be a hosting control panel with a file manager, an FTP client like FileZilla, SSH access to the server, or a project repository from which deployment happens. Check that your login and password are current and that you can actually create a file in the site's root folder.
2. A text editor. Any editor that saves clean, unformatted text will do: Visual Studio Code, Sublime Text, Notepad++ on Windows, TextEdit in plain text mode on Mac. Word and Google Docs won't work: they add invisible characters and smart quotes.
3. A list of your site's pages. Export it from your sitemap.xml, from your CMS admin panel, or from any crawler like Screaming Frog. A simple table with each page's URL and title is enough.
4. Access to analytics. Yandex Metrica, Google Analytics, or server logs will help you understand which pages actually matter to users.
5. A browser and the curl utility. Curl comes preinstalled on Mac and Linux, and it's available from the command line on Windows 10 and 11. You'll need it to check server response headers.

System Requirements

None. The llms.txt file is just plain text. It doesn't need PHP, a database, or any special module on the server. The only requirement: your server must serve static files from the domain root — and any hosting can do that.

What to Download and Install

• A code editor, if you don't have one yet. We recommend Visual Studio Code: it's free, highlights Markdown, and shows invisible characters.
• An FTP client, if your hosting panel doesn't have a file manager.
• Optional: Python 3 and the llms-txt package for generating context from your finished file. It comes in handy in the advanced section — not required for basic setup.

Backups

Creating llms.txt doesn't modify any existing site files, so the risk is minimal. However, during the server header setup step, you'll be editing configuration — for example, the .htaccess file or an Nginx config. Before that, be sure to download the current version of the config file to your computer and save it with the date noted. If the site stops loading after your edit, you'll just restore the old version.

Tip: Set up a separate project folder on your computer — something like site-llms. Keep the page table, file drafts, and config copies in it. Six months from now, when it's time to update llms.txt, you'll thank yourself.

Key Concepts: What llms.txt Is and How It Works

Before creating the file, let's go over a few terms in plain language. It'll take five minutes, and everything else will make sense after that.

What Is llms.txt

llms.txt is a text file in Markdown format placed in a site's root. It contains a short project description and a structured list of links to key pages with explanations. The name reads as "L-L-M-S-dot-t-x-t": LLMs stands for Large Language Models.

The standard was proposed in the fall of 2024 by developer Jeremy Howard from Answer.AI. The idea was simple: language models have a limited "attention window" and struggle to parse heavy HTML pages with menus, ads, scripts, and popups. The llms.txt file gives them a clean content map without the noise. Since then, hundreds of companies have adopted the format — especially in documentation and development — and by 2025-2026, it became one of the baseline elements of preparing a site for AI.

What llms.txt Is Not

This is important to understand right away so you don't confuse the tools.

• It's not a permissions file. It doesn't block or allow anything. Bot access rules live in a different file, and we have separate articles about that on our blog. We won't touch on it here.
• It's not a sitemap in the sitemap.xml sense. A sitemap lists every URL for indexing. The llms.txt file, by contrast, curates the most important content and explains the purpose of each link.
• It's not a guarantee of appearing in AI answers. The file helps models understand your content, but it doesn't force them to cite it.

Two Files: llms.txt and llms-full.txt

The standard proposes two entities:

• llms.txt — brief navigation. A title, an annotation, sections with links. Typically 20 to 200 lines.
• llms-full.txt — the full content of key pages in one text. A model or tool can load it entirely without following links.

The first file is required, the second is desirable. We'll create both.

What the File Structure Looks Like

The specification defines a strict order of blocks:

1. A level-one heading with the project name. It's the only required element.
2. A blockquote block with a brief description: one to three sentences about what the site is and who it's useful for.
3. Optional paragraphs with details: features, limitations, context.
4. Level-two sections, each with a list of links. Line format: dash, link title in square brackets, URL in round brackets, colon, and a short description.
5. An optional section named Optional. The model can skip links in it if it's low on context space.

Here's a minimal example we'll keep coming back to:

# Project Name

> One or two sentences: what this is and who it's for.

## Documentation

- [Quick Start](https://example.com/docs/start): how to get going in 10 minutes
- [Pricing](https://example.com/pricing): prices and limits

## Optional

- [Changelog](https://example.com/changelog): what's been updated

The line break symbol here is just illustrative; in a real file, it's simply new lines. Looks simple enough, right? The real work isn't in the syntax — it's in choosing pages and writing descriptions. That's what we'll tackle next.

Who Reads llms.txt in 2026

Honest answer: not everyone, and not uniformly. The format is actively used by AI developer tools, code editors with AI assistants, some AI search engines and agents that hit a site at the moment a user asks a question. The biggest search companies haven't officially confirmed support. Still, the standard has become a common language in the industry, the file costs nothing to maintain, and its absence certainly doesn't help. Plus there's a side benefit: by creating llms.txt, you clean up your own content.

Step 1: Content Audit and Choosing Pages for AI

Goal of this stage: get a table of 10-50 pages that will go into llms.txt, each tagged with a section and a draft description.

The most common beginner mistake is stuffing everything into the file. The model gets a thousand lines, can't find what matters, and performs worse than without the file. Your job is to select what genuinely explains the product and answers your audience's questions.

Instructions

1. Open the page list table you prepared earlier. If you don't have one, open your-domain/sitemap.xml in a browser and copy the URLs into a spreadsheet.
2. Add columns: "Title," "Section," "Description for AI," "Priority."
3. Go through each row and answer: "If someone asks an AI about my product, will this page help give an accurate answer?" If yes, mark priority 1. If the page is useful but secondary, mark 2. Everything else gets a zero.
4. Be sure to include as priority 1: the main product description page, pricing page, About page, documentation or how-to sections, contact page, and terms of use. For an online store — category pages and shipping info. For a media-buying or affiliate project — the program terms page and offer description.
5. Open your analytics. Look at the 20 most-visited pages over the last three months. If any of them aren't in priority 1, reconsider: users consider them important.
6. Exclude utility pages: cart, account dashboard, search results, pagination pages, UTM-tagged duplicates, outdated promotions.
7. Group priority 1 and 2 pages into three to six sections. Typical names: "Product," "Documentation," "Pricing," "Blog," "Support," "Case Studies." Write the section name in the "Section" column.
8. For each selected page, write a draft description of 8-15 words. Don't copy the meta description: that's written for people in search results. Write so the model understands when this page is worth opening. Bad: "Our pricing." Good: "Mobile proxy pricing by country, traffic limits, and IP rotation terms."

Tip: Imagine the descriptions are being read by a new support agent who needs to understand in a minute where to send a customer. If your description is enough for them, it's good enough for a language model too.

Expected Result

You have a table with priority 1 and 2 pages filtered, each assigned to a section and with a draft description. Typically that's 10-50 rows for a standard site and up to 200 for large documentation.

Verification: Read only the "Description for AI" column top to bottom without looking at the URLs. If the descriptions make it clear what the site does and how the product works, the audit is done right.

Potential Issues

• Too many priority 1 pages. If you have more than 60, you're not being strict enough. Ask yourself again: without which page can the model not answer correctly? Move the rest to priority 2 or the Optional section.
• Content is scattered, no clear sections. This signals problems with your site structure. For llms.txt, just create logical groups even if they don't exist in the menu. You can sort out navigation later.
• Pages are blocked from indexing or only accessible after login. Don't include them. The model won't be able to read them anyway, and you'll be giving it false links.

Step 2: Creating the llms.txt File and Its Skeleton

Goal of this stage: create a file with the correct name and encoding, fill in the title, annotation, and context paragraphs.

Instructions

1. Open your text editor. Create a new file via the "File" menu and "New File," or with Ctrl+N.
2. Save it right away: "File" menu, "Save As." In the filename field, type exactly llms.txt in lowercase. Check that the editor didn't add a second extension like llms.txt.txt. On Windows, enable file extension display in File Explorer for this.
3. Make sure the file encoding is UTF-8 without BOM. In Visual Studio Code, the encoding is shown in the bottom right corner; click it, select "Save with Encoding," and choose UTF-8. In Notepad++, open the "Encoding" menu and select "Encode in UTF-8" without the BOM label.
4. Write the level-one heading as the first line: hash symbol, space, project name. For example: # MobileProxy.space. Use the brand name as customers know it — no slogans.
5. Leave one blank line.
6. Write the blockquote block: greater-than symbol, space, and one to three sentences about what the project is. Answer "what is it," "who is it for," "what's different." Example: > Mobile proxy rental service with carrier IPs for marketers, media buyers, and developers. Supports address rotation via link and API, works with anti-detect browsers and scraping tools.
7. Leave a blank line.
8. Add one to three regular paragraphs with context that helps the model avoid mistakes. Good candidates: operating regions, site languages, what the service doesn't do, date the info was last accurate. Example: "Prices are in USD and updated monthly. The site is available in English and Spanish; the Spanish version is located in the /es/ subfolder. The service provides infrastructure and does not offer advertising campaign setup services."
9. Save the file with Ctrl+S.

Caution: The file must have exactly one level-one heading, and it must be the first line. If you put a comment before it, a blank line with spaces, or a second heading, tools that strictly follow the specification may refuse to parse the file.

How to Write an Annotation That Works

The blockquote block is the most-read part of the file. It's what the model most often uses when summarizing what your site is about. A few rules:

• No evaluative words. "Best," "unique," "number one" carry no information and reduce the model's trust in the rest of the text.
• Specifics over abstractions. Not "solutions for business," but "mobile proxy rental with IPs from US and European carriers."
• State the target audience explicitly. The model matches it against the user's question.
• Stay within 300-400 characters. Longer, and the model may truncate it.

Tip: Write three versions of your annotation and paste each into any available AI assistant with the question: "What does this company do and who needs it? Answer in one sentence." Pick the version where the assistant answered most accurately.

Expected Result

The llms.txt file exists on disk, saved in UTF-8, starts with a single heading, contains an annotation in the blockquote block, and one to three context paragraphs. No link sections yet — that's the next step.

Verification: Open the file in a browser by dragging it into the window. You should see clean text with no garbled characters where Cyrillic would be. If you see question marks or diamonds instead of letters, the encoding is wrong — go back to step 3.

Potential Issues

• The editor automatically replaced straight quotes with "smart" ones or hyphens with em dashes. Turn off autocorrect in settings or use a code editor. In Markdown links, such characters break the formatting.
• The file saved as llms.txt.txt. Rename it via File Explorer or Finder, after enabling extension display.

Step 3: Filling In Sections and Links

Goal of this stage: move the selected pages from your table into the file with correct syntax, group them by section, and add the Optional block.

Instructions

1. Open llms.txt and your table from step 1 side by side.
2. After the context paragraphs, leave a blank line and write a level-two heading for the first section: two hash marks, space, name. For example: ## Product.
3. Leave a blank line.
4. For each page in this section, write a list line strictly following the template: dash, space, opening square bracket, page title, closing square bracket, opening round bracket, full URL with protocol, closing round bracket, colon, space, description. Example: - [Mobile Proxy Pricing](https://example.com/pricing): cost by country and carrier, traffic limits, IP rotation terms.
5. Use absolute URLs starting with https. Relative paths like /pricing can't be resolved by many tools because they read the file outside a browser context.
6. Take the link title from the page's heading, but shorten it to five to eight words. Take the description from the "Description for AI" column.
7. Repeat steps 2-6 for each section. Order sections from most important to least: product and pricing first, then documentation, then blog and case studies.
8. Within a section, also order pages by importance. The first three links in each section get read the most.
9. Add ## Optional as the last section. Move priority 2 pages there: blog archive, changelog, secondary case studies, careers page. The section name must be written in Latin letters exactly as: Optional. It's a reserved word in the specification.
10. Save the file.

Tip: If your site has pages that already serve content in clean form — for instance, Markdown versions or plain text documentation pages — link to those instead of HTML versions. The model gets text without menus and scripts and understands it better.

Example of a Finished Section

## Documentation

- [Setting Up a Proxy in an Anti-Detect Browser](https://example.com/docs/antidetect): step-by-step profile setup, IP check, and common errors
- [IP Rotation API](https://example.com/docs/api): methods, request parameters, limits, and JSON response examples
- [Connection Formats](https://example.com/docs/formats): HTTP, SOCKS5, login-based and IP-based authentication

How to Write Link Descriptions

The description after the colon is a hint to the model about when to open the page. The more precise the hint, the better the model chooses the source for its answer. The rules are simple:

• Answer "what's inside," not "why read it." Not "useful article about proxies," but "comparison of mobile, residential, and datacenter proxies by speed, cost, and block risk."
• Use the words people type into their questions. If customers ask "how to change my IP," the description should say "IP rotation."
• Don't repeat the link title. The title says "what it is," the description says "what's specifically there."
• Stay within one line. Line breaks inside a list item break parsing.

Expected Result

The file contains three to six level-two sections, each with 2 to 20 links with descriptions, and an Optional section at the end. Total length: 20 to 200 lines.

Verification: Paste the file content into any online Markdown viewer or open preview in your code editor. All links should become clickable, headings should be larger, lists should have bullets. If any line shows as plain text with brackets, there's a syntax error in it.

Potential Issues

• The link didn't become clickable. Most often, the space after the dash is missing, a bracket is skipped, or there's a space between the square and round brackets. There should be no space there.
• The URL has spaces or non-Latin characters. Encode the URL: a space becomes %20, non-Latin characters become their percent representations. Easier: copy the URL from the browser's address bar — it's already encoded.
• The description contains a colon. The first colon after the closing round bracket is treated as the separator; subsequent colons inside the description are allowed, but it's better to rephrase to avoid confusion.

Step 4: Creating llms-full.txt and Clean Text Versions of Pages

Goal of this stage: compile an extended file with the full text of key pages so tools can load all the content in one request.

This step is optional, but it's exactly what gives the biggest payoff for documentation, instructions, and detailed product descriptions. If llms.txt is a table of contents, then llms-full.txt is the entire book.

Instructions

1. Create a new file and save it as llms-full.txt in the same UTF-8 encoding.
2. Copy the level-one heading and blockquote block from llms.txt to the beginning. The files should start the same way.
3. For each priority 1 page, open it in the browser and copy the main text: headings, paragraphs, lists, tables. Don't copy menus, footers, subscription forms, comments, or ad blocks.
4. Paste the text into the file under a level-two heading with the page name. Downgrade the internal page headings by one or two levels: whatever was a level-two heading on the site becomes a level-three in the file. This preserves the hierarchy.
5. After each block, add a line with the source URL, for example: Source: https://example.com/docs/api. This helps the model reference the specific page.
6. Separate page blocks with a blank line and a line of three dashes — the standard Markdown horizontal rule.
7. Convert tables to Markdown format with vertical bars, or turn them into lists if the structure is simple.
8. Remove utility phrases like "click here," "read more," "share." They're meaningless in text without an interface.
9. Save the file.

Caution: Don't include customer personal data, internal documents, coupons, or any information you wouldn't want a stranger to see an AI assistant repeat. Everything in this file is essentially published.

How Much Text Should There Be

A reasonable range is 20,000 to 300,000 characters. Less, and the file doesn't justify its existence — llms.txt alone suffices. More, and many tools won't load it in full. If you have a lot of content, create several thematic files, for example llms-full-docs.txt and llms-full-blog.txt, and reference them from llms.txt in a separate section.

Tip: For WordPress sites and most CMS platforms, there are plugins and modules that generate llms.txt and llms-full.txt automatically from published posts. Search for them in the extension directory using "llms txt." Autogeneration saves hours, but always check the result manually: plugins often pull everything in, including tag pages and drafts.

Expected Result

The llms-full.txt file contains the same heading and annotation as llms.txt, followed by the full clean text of key pages with source references.

Verification: Open the file and read a random fragment from the middle. If it's clear which page is being discussed and the text reads without interface clutter, you're good. If you see fragments like "Menu Home Pricing Contact," clean them out.

Potential Issues

• Copying from a browser loses heading structure. Copy in parts and set hash marks manually, or use a browser extension to save the page as Markdown.
• The file ended up several megabytes. Split it into thematic parts or keep only priority 1 pages.

Step 5: Uploading Files to the Server

Goal of this stage: place llms.txt and llms-full.txt in the site's root so they're accessible at your-domain/llms.txt.

The method depends on how your site is built. Below are four typical scenarios. Pick yours.

Option A: Hosting Panel with File Manager

1. Log into your hosting control panel using the login and password from your provider's email.
2. Find the section named "File Manager," "Files," or "File Browser."
3. Navigate to the site's root folder. It's usually called public_html, www, htdocs, or bears the domain name. Landmark: it contains an index.html or index.php file and an existing sitemap.xml.
4. Click the "Upload" button. A file selection window opens.
5. Select llms.txt and llms-full.txt from your computer and confirm the upload. Wait for the completion indicator.
6. Make sure the files appear in the list next to the index file, with permissions set to 644 — read for everyone, write only for the owner. Panels usually set these permissions automatically.

Option B: FTP Client

1. Open FileZilla or a similar client.
2. In the top panel, enter the host, username, password, and port from your host's email. Click "Quickconnect."
3. On the right side of the window, find the site's root folder using the same landmarks as above.
4. On the left side, find the folder with your files on your computer.
5. Drag llms.txt and llms-full.txt from left to right. A transfer queue appears at the bottom of the window; wait until it empties.
6. Right-click the uploaded file, select "File Permissions," and check the value is 644.

Option C: Site Built from a Repository with Auto-Deploy

1. Identify the folder from which static files are published. In most frameworks, it's called public or static and sits in the project root.
2. Place llms.txt and llms-full.txt in this folder next to the favicon and robots file.
3. Commit the changes and push them to the deploy branch.
4. Wait for the build to finish. Time depends on the project — usually one to ten minutes.

Option D: Website Builder

This is trickier: not every builder lets you add an arbitrary file to the domain root. Here's the sequence:

1. Open your site settings and look for a section like "Files," "File Uploads," "SEO," or "Advanced."
2. If you can upload a file with a specified path, specify the path /llms.txt.
3. If you can't, but there's a redirect or rules setting, create a rule: requests to /llms.txt redirect with code 200 or 301 to the uploaded file. Some platforms call this "forwarding" or "routing rules."
4. If that's also unavailable, contact the builder's support with a request to host a text file at the root. Phrasing: "I need to host a static llms.txt file at domain/llms.txt with content type text/plain."

Caution: The file must be in the domain root, not in a subfolder. A URL like domain/files/llms.txt won't be found by tools because they look for the file at a fixed path, like favicon or sitemap.

Tip: If your site runs on multiple domains or subdomains — for example, the main domain and a separate documentation subdomain — place llms.txt on each one. Each domain's file describes only its own content, and in the link section can point to the neighboring domain.

Expected Result

Typing your-domain/llms.txt into the browser's address bar opens your text. Same for llms-full.txt.

Verification: Open both URLs in incognito mode to rule out browser cache. You should see the file's text as-is: the heading with a hash mark, the quote with a greater-than sign, the list with dashes. If the browser offers to download the file instead of displaying it, that's not critical — we'll fix it in the next step.

Potential Issues

• 404 error. The file isn't in the root or the name has a typo. Check case: LLMS.txt and llms.txt are different files on Linux servers.
• 403 error. Wrong permissions. Set 644 via the file manager or FTP.
• Old version opens after updating. Hosting or CDN cache is at play. Clear the cache in the panel or wait for the cache lifetime to expire.
• Non-Latin characters display incorrectly. The server isn't reporting the encoding. This is fixed in the next step.

Step 6: Configuring Server Response Headers

Goal of this stage: make the server serve the file with the correct content type and encoding, allow access from browser-based AI agents, and not cache the file for too long.

Not everyone needs this step. If the check in step 5 showed correct text, you can just use the verification below. But correct headers increase the chance that tools process the file without surprises.

Which Headers You Need

• Content-Type: text/plain; charset=utf-8 or text/markdown; charset=utf-8. The first option is more universal.
• Cache-Control: max-age=3600 — one-hour caching. Enough to ease server load, but updates are picked up quickly.
• Access-Control-Allow-Origin: * — permission to read the file from web apps on other domains. Some AI tools work directly in the browser and can't load the file without this header.

Instructions for Apache via .htaccess

1. Download the current .htaccess file from the site root to your computer and save a copy. If it doesn't exist, create a new empty one.
2. Open the file in your editor and add the following block at the end:

<Files "llms.txt">
 Header set Content-Type "text/plain; charset=utf-8"
 Header set Cache-Control "max-age=3600"
 Header set Access-Control-Allow-Origin "*"
</Files>
<Files "llms-full.txt">
 Header set Content-Type "text/plain; charset=utf-8"
 Header set Cache-Control "max-age=3600"
 Header set Access-Control-Allow-Origin "*"
</Files>

1. Save and upload the file back to the site root, replacing the old one.
2. Immediately open your site's homepage. If it loads, great. If you see a 500 error, immediately restore the saved copy: your hosting likely doesn't have the headers module enabled. In that case, contact hosting support to enable mod_headers.

Instructions for Nginx

1. Connect to the server via SSH.
2. Back up the site's config file — it's usually in /etc/nginx/sites-available/.
3. Inside the server block, add:

location = /llms.txt {
default_type text/plain;
charset utf-8;
add_header Cache-Control "max-age=3600";
add_header Access-Control-Allow-Origin "*";
}
location = /llms-full.txt {
default_type text/plain;
charset utf-8;
add_header Cache-Control "max-age=3600";
add_header Access-Control-Allow-Origin "*";
}

1. Test the configuration with nginx -t. You should see a success message.
2. Reload the configuration with systemctl reload nginx.

Instructions for Website Builders and Cloud Platforms

Look for a "Headers" or "Custom Headers" section in settings. Many platforms support a headers-rules file in the project root; syntax differs, but the point is the same: set Content-Type and Cache-Control for the /llms.txt path. If that's not possible, don't worry: most servers serve .txt as text/plain by default, which is already acceptable.

Expected Result

The server serves the file with a 200 code and the right headers.

Verification: Open the command prompt or terminal and run curl -I https://your-domain/llms.txt. The response should show HTTP/2 200 or HTTP/1.1 200 OK, a content-type: text/plain; charset=utf-8 line, and a cache-control: max-age=3600 line. If you want to see the content too, drop the -I flag.

Potential Issues

• 500 error after editing .htaccess. Restore the copy, check there are no stray characters left, and check with your host whether the headers module is enabled.
• Headers didn't change. A CDN or proxy cache in front of the server is serving the old version. Clear the CDN cache in its panel.
• Content-Type is still text/html. Some rule rewrites all requests to index.php. Make sure the rule excludes existing files — usually via an "if file doesn't exist" condition.

Verification: llms.txt Readiness Checklist

You've completed all the steps. Now let's make sure the result actually works, not just "the file is there." Go through the checklist and tick things off.

Checklist

1. The domain/llms.txt URL opens in a browser in incognito mode and shows the text.
2. The first line of the file is the sole level-one heading with the project name.
3. Right after the heading comes the blockquote block with an annotation of up to 400 characters.
4. All links are absolute, start with https, and open without redirects or errors.
5. Every link has a description after the colon.
6. Sections are sorted by importance, with secondary content moved to Optional.
7. Non-Latin characters display correctly, encoding is UTF-8.
8. The curl -I command shows a 200 code and type text/plain or text/markdown.
9. If llms-full.txt was created, it opens at its own URL and starts the same way as llms.txt.
10. The files contain no personal data or internal information.

How to Really Test It

1. Link check. Copy all URLs from the file into any bulk link checker or server response tool. All should return 200. One broken link won't break the file, but it reduces the model's trust.
2. Validity check. Paste the content into an online llms.txt validator; by 2026 there are several such services — search by the standard's name. The validator will show whether it recognized the heading, annotation, and sections.
3. Real model test. Open any AI assistant that can read links. Give it your file's URL and ask: "Study this file and tell me what the company does, how much the service costs, and where to find setup instructions." If the answer is accurate and the assistant references the right pages, you've nailed it. If it stumbles, see which descriptions confused it and clarify them.
4. Test from different networks. If you have access to mobile proxies in different regions, open the file through them. This shows whether the server serves different content for different countries and whether bot protection blocks access to the file for certain IP ranges. AI agents come from all sorts of addresses, and the file must be accessible to all.

Success Indicators

• All ten checklist items are complete.
• An AI assistant summarized your site's essence without factual errors.
• Two to four weeks later, server logs show requests to /llms.txt from AI company agents. We'll explain how to find them in the advanced section.

Common llms.txt Mistakes and How to Fix Them

Here are the problems almost everyone hits when creating the file for the first time. Format: problem, cause, solution.

1. The File Opens, But the AI Tool Says It Can't Find It

Cause: Bot protection at the hosting or CDN level shows a captcha or JavaScript check to anything that looks like an automated request. The browser passes the check invisibly, but the agent doesn't.

Solution: In the protection settings, add an exception for the /llms.txt and /llms-full.txt paths. Verify the result with a curl command without browser headers: if curl gets text rather than an HTML check page, it's fixed.

2. The Model Summarizes Your Site Incorrectly Despite the File Existing

Cause: The annotation is vague, or link descriptions just repeat titles and carry no information.

Solution: Rewrite the annotation following the rules from step 2 and the descriptions following step 3. Run the assistant test again.

3. The File Has Hundreds of Links and Became Useless

Cause: Plugin autogeneration or perfectionism: "let's include everything."

Solution: Go back to the audit from step 1. Keep no more than 50 links in the main sections, move the rest to Optional or remove them. Completeness is what llms-full.txt provides, not the length of the table of contents.

4. Non-Latin Characters Turned Into Question Marks

Cause: The file was saved in Windows-1251 encoding, or the server isn't sending the charset.

Solution: Resave the file in UTF-8 without BOM and add charset=utf-8 to the Content-Type header following the instructions from step 6.

5. The File Was Updated, But AI Sees the Old Version

Cause: Long caching on the CDN or on the tool's side.

Solution: Set Cache-Control with max-age to no more than an hour, clear the CDN cache. Tools refresh their copies at different intervals — give them a few days.

6. Links Point to Pages with Redirects

Cause: The file lists URLs without a trailing slash or with http instead of https, and the server redirects.

Solution: Open each link and copy the final URL from the address bar. Replace it in the file. Agents don't always follow redirects.

7. The Level-One Heading Isn't First in the File

Cause: A blank line with spaces, an invisible BOM character, or a comment ended up before it.

Solution: In your code editor, enable invisible character display, delete everything before the first hash mark, and save without BOM.

8. The File Describes Several Sites at Once

Cause: Wanting to save effort and reference all the company's projects from one file.

Solution: One domain — one file about that domain. You can reference other projects in a separate section, but the annotation and main sections must be about the current site.

Advanced Capabilities: Taking llms.txt Further

Basic setup is done. This section is for those who want to squeeze the most out of the file: automate updates, monitor AI bots, and use llms.txt as part of a promotion strategy.

Automatic Generation from CMS or Build

For sites with frequently changing content, manual file updates get old fast. Automation options:

• CMS plugin. WordPress, Joomla, and other systems have modules that build llms.txt from selected post types. Configure them to include only the right categories and pull descriptions from a dedicated field rather than the first lines of text.
• Build-time script. For static sites and frameworks, write a small script that reads page metadata — for example, title and llm_description fields in Markdown file front matter — and assembles llms.txt from them. Run it before deployment.
• Separate Markdown versions of pages. Some sites serve each page in both HTML and Markdown at a URL with a .md suffix. Then llms.txt can link to the clean versions, and llms-full.txt can be assembled by simple concatenation. This is the most convenient format for models.

Validating the File with Command-Line Tools

There's an official Python package called llms-txt. It turns the file into context for a model and validates the structure as a bonus. Install and run:

pip install llms-txt
llms_txt2ctx https://your-domain/llms.txt

If the command outputs structured text with sections, the file parsed correctly. If it threw an error, check which line it tripped on: usually it's a bad link syntax.

Monitoring AI Bots in Logs

The most honest way to find out whether your file is being read is to check the web server logs. Here's the procedure:

1. Find the access log file. In Nginx, it's usually /var/log/nginx/access.log; in Apache, it's access.log in the site's logs folder. On shared hosting, logs are available in the panel under "Statistics" or "Logs."
2. Filter lines containing llms.txt. In the terminal: grep llms.txt access.log.
3. Look at the User-Agent field in the matching lines. AI company agents usually identify themselves with recognizable names containing the company name or words like Bot, Agent, User.
4. Set up a simple table: date, agent, file. Once a month, review who's coming and how often. Growth in request count is a good sign.

Tip: If you want to see your site "through the eyes" of an agent coming from another country, use a mobile proxy from the region you need and make the request through it: curl -x proxy-address https://your-domain/llms.txt. This verifies that CDN geo-settings, regional redirects, and bot protection don't prevent the file from being retrieved from abroad. This is especially relevant for projects operating in multiple countries.

llms.txt for Landing Pages and Affiliate Projects

Media buyers and small landing page owners often think they don't need the file: they have few pages. In practice, even for a one-pager, llms.txt is useful: it gives the model a precise formulation of your offer, terms, geography, and a link to the rules. When a user asks an assistant about the product, it takes facts from your file rather than making them up. For a network of landing pages, make a shared template and fill in the name, annotation, and links automatically.

Connection to Generative Optimization

The llms.txt file is one element of what's being called generative optimization in 2026: working to make your brand appear correctly and frequently in AI answers. Other elements include structured data on pages, clear answers to questions at the start of articles, authorship and update dates, and consistency of company facts across all sources. The file ties it all into one map. Update it with every significant product or pricing change, and models will get an up-to-date picture.

Versioning and Freshness Date

Add a line with the last update date to a context paragraph, for example: "Information is current as of March 2026." This helps the model assess data freshness. Keep the file in version control alongside your site's code so you can see the change history.

FAQ: Common Questions About llms.txt

Is llms-full.txt Mandatory?

No. Only llms.txt is required. But if you have documentation, instructions, or detailed descriptions, the full version significantly improves the quality of answers from tools that use it. For a five-page landing site, one file is enough.

Can I Write the File in a Non-English Language?

Yes. Use the language of your site's primary content. If your site is bilingual, state in the context paragraph where the second version lives, and optionally create a separate file on a subdomain or in a language folder, referencing it from the main one. Section names can also be in your language, except for the reserved word Optional.

How Often Should I Update llms.txt?

With every pricing change, new section launch, terms change, or page removal. If nothing significant happens, a quarterly review is enough: check links, refresh descriptions, update the date.

Do I Need to List llms.txt in the Sitemap or Register It Somewhere?

No. Tools look for the file at the standard root path, like favicon. There's no special registration. Some public directories collect sites with llms.txt; adding yours is optional but won't hurt.

Will the File Affect My Regular Search Rankings?

No direct impact: the file isn't indexed as a page and isn't a ranking factor. Indirectly it helps: it cleans up the structure, and accurate mentions in AI answers drive branded traffic.

What If My Site Builder Won't Let Me Put the File in the Root?

First, look for a file upload setting with an arbitrary path or redirect rules. If there's nothing, write to the builder's support: by 2026, a request to host llms.txt is nothing new to them. The last resort is hosting the file on a separate subdomain and linking to it from the About page, though the standard path is preferable.

Should I Include Blog Pages in the File?

Include articles that answer frequent customer questions and stay relevant: how-tos, comparisons, breakdowns. News and promo announcements are better left out or moved to Optional. For a large blog, make a separate section with the 10-20 most useful pieces.

Can I Use One File for Several Company Domains?

No. Each domain is described by its own file. Cross-links between projects are acceptable in a separate section, but the annotation and main sections must relate to the current domain.

How Do I Know AI Is Actually Using the File?

Check the logs following the advanced section's instructions and periodically ask assistants about your product. If the wording in their answers matches your annotation and the assistant names the right pages, the file is working. Keep a monthly check log to track the trend.

What If the Site Stops Loading After a Server Config Edit?

Immediately restore the saved config file copy and verify the site is back up. Then contact hosting support about enabling the needed modules. Meanwhile, llms.txt will keep working without extra headers: servers serve .txt as plain text by default.

Conclusion

Let's sum up what you've done. You audited your content and selected the pages that truly explain your product. You created an llms.txt file with a clear heading, an honest annotation, and structured link sections. You compiled an extended llms-full.txt version with the full text of key pages. You uploaded the files to the site root, configured server headers, and verified the result with curl, a validator, and a real AI assistant. Now language models get not a chaos of HTML pages from your site, but a clear map with explanations.

What to Do Next

• Set a reminder to review the file in three months. Check links, update prices and the freshness date.
• Once a month, check the logs and note which agents are reading llms.txt. It's free analytics on AI's interest in your project.
• Ask several assistants questions about your product and compare the answers with the facts. Discrepancies are a signal to clarify descriptions in the file or on the pages themselves.

Where to Grow From Here

The llms.txt file is the first step in working with AI visibility. Next: structured data on pages, clear question-answering at the start of every article, Markdown versions of key materials, consistent company facts across all directories and catalogs. If you operate in multiple regions, add regular content availability checks via mobile proxies in the countries you need: agents come from all sorts of addresses, and the file must be accessible to every one of them. The more transparent and structured your content, the more accurately AI will tell users about you. Which means more people will come to you already informed and with the right expectations.