Decoding

Free llms.txt generator

Optimize how Gen AI and Language Models (LLMs) understand the content of your website with Decoding’s new free tool, the LLMs.txt Generator, which allows you to add the most important content of your website and download a structured text file.

The llms.txt file is a proposed web standard, not official or required, but might help LLMs process websites’ content with more context.



Block specific AI crawlers (via dedicated sections)










Instructions

Here’s how to use the llms.txt Generator on the page:

  1. Pick a default policy
    • Allow all: You’re fine with AI/LLM crawlers using your content.
    • Disallow all (default): You don’t want AI/LLM crawlers to use your content.
    • Custom: You’ll specify exactly which parts are allowed/disallowed below.
  2. Set path rules (one per line)
    • Allow paths: Enter folders or pages you want to allow (e.g., /public, /press-kit).
    • Disallow paths: Enter folders or pages you want to block (e.g., /private, /drafts, /api/).
    • Tips:
      • Paths should start with a / (root-relative).
      • A trailing / usually means “this folder and everything inside it.”
  3. (Optional) Set Crawl-Delay
    Enter a number of seconds to slow compliant crawlers (e.g., 10). Leave blank for no delay.
  4. Choose specific AI crawlers to target
    Tick any bots you want dedicated sections for (e.g., GPTBot, Google-Extended, ClaudeBot, PerplexityBot).
    • If your default policy is Disallow, selected bots get Disallow: /.
    • If Allow, selected bots get Allow: /.
    • If Custom, they mirror your path rules.
  5. (Optional) Add Sitemaps
    Paste full URLs, one per line (e.g., https://example.com/sitemap.xml).
  6. (Optional) Add Contact & License
    • Contact: Email or URL where people can reach you.
    • License: A URL describing your content usage terms.
  7. (Optional) Notes
    Add any comments (they’ll appear as # comments at the top of the file).
  8. Generate the file
    The “Generated llms.txt” box updates automatically as you type.
  9. Export your file
    • Copy: Click Copy, then paste into a new file named llms.txt.
    • Download: Click Download llms.txt to save it directly.
  10. Publish it on your site
    • Upload llms.txt to the root of your site (the top level).
      • Example: it should be reachable at https://yourdomain.com/llms.txt.
      • For a subdomain, use that host’s root (e.g., https://blog.yourdomain.com/llms.txt).
  11. Verify it’s live
    • Visit the URL in your browser to confirm it loads.
    • If you use a CDN or cache, purge/clear it.
  12. (Recommended) Mirror key rules in robots.txt
    Some crawlers mainly honor robots.txt. Consider adding matching user-agent blocks there (e.g., a User-agent: GPTBot section) so your intent is clear.

Quick tips & gotchas

  • Keep paths simple (no wildcards needed).
  • If you choose Disallow all, but want some public areas open, list them in Allow paths (e.g., /public).
  • Don’t accidentally block your sitemap if you want crawlers to find it.
  • Adoption of llms.txt varies; pairing with robots.txt increases coverage.
  • You can return anytime, adjust settings, and re-download a fresh llms.txt.

What is LLMs.txt?

llms.txt is a new proposed web standard, just like robots.txt, but designed for for AI consumption, so models like ChatGPT, Perplexity, or Claude can quickly summarize or generate content based on your website.

This file will help AI systems understand and give more context and to attribute correctly your content in LLM and Gen-AI search answers.

LLMs.txt is a concise, markdown-formatted, summary of your website’s most important content, including internal links.

Why is this file important?

LLMs and GenAI-powered search have a small window of context to handle the complexity of large websites, the llms.txt helps give quickly that context with the most important and relevant content.

Can Google or other search engines use my LLMs.txt?

Currently, search engines don’t use LLMs.txt. However, as AI integration grows, it’s possible that search engines might leverage this LLMs.txt files in the future.