llms.txt Generator
Discover your site's pages, curate your most important ones, and export a properly formatted llms.txt and llms-full.txt — ready to publish at the root of your domain.
What is llms.txt?
llms.txt is a plain-text Markdown file placed at /llms.txt on your domain. Proposed in 2024 by fast.ai's Jeremy Howard as a lightweight complement to robots.txt and sitemap.xml, it's a curated, LLM-readable index of a site's most important pages — written to be parsed by AI systems that want to understand your site's structure and content without crawling every page individually.
Where robots.txt is restrictive (telling crawlers what they can't access) and sitemap.xml is exhaustive (listing every URL), llms.txt is descriptive and curated: it says "here's what we are, here are the 10–30 pages that matter most, and here's a one-line summary of each." AI answer engines like ChatGPT, Perplexity, and Claude can use this file to quickly understand your site without the noise of a full crawl — making it more likely your content is accurately represented in AI-generated answers.
llms.txt vs llms-full.txt
llms.txt— Curated indexThe standard file: 10–30 of your most important pages, hand-picked and organised into sections (Docs, Guides, etc.). This is what most AI systems read. Short, deliberate, and maintained by humans. Generated by this tool from your selected pages.
llms-full.txt— Complete inventoryThe extended version: every page discovered in your sitemap in the same Markdown format, grouped under a single 'All pages' section. Useful for sites that want to expose their full content inventory to AI systems that support it. Larger projects sometimes generate this automatically from their CMS.
How this generator works
Sitemap-first discovery
We start by checking your robots.txt for a Sitemap: directive — the most authoritative source for your sitemap's location. If not found, we try the default /sitemap.xml. We handle both standard sitemaps (urlset) and sitemap indexes (sitemapindex) that point to multiple child sitemaps.
Smart sorting
Pages are sorted by their declared <priority> value (descending), then by <lastmod> date. This surfaces your most important pages first — the ones your own sitemap says matter — rather than crawling in document or discovery order.
Metadata enrichment (first 40)
We fetch the title and meta description of your top 40 pages to pre-populate the editor. Pages beyond the first 40 appear as bare URLs you can add manually. This limit exists to prevent this tool from becoming a scraper on large sites — and because a good llms.txt shouldn't have hundreds of pages anyway.
You curate, we format
The editor gives you editable title and description fields for each page, section assignment, and a live preview of the generated Markdown. The generator produces a draft; good llms.txt files are written by humans who know what their site is actually about.
Already have an llms.txt?
Check your full AI readiness score across 6 factors — robots.txt crawler access, llms.txt, JSON-LD schema, SSR content, metadata, and sitemap.
Frequently asked questions
Want automatic bot protection instead of manual files?
Sign up free — connect your site in minutes, no commitment, no CAPTCHA friction added during evaluation.