AI Readiness Checker
Score your site across 6 AI-readiness factors: crawler access, llms.txt, structured data, server-side content, metadata, and sitemap hygiene.
What is AI readiness?
AI readiness describes how well your website can be discovered, understood, and cited by AI answer engines — ChatGPT, Perplexity, Claude, Google AI Overviews, and the dozens of LLM-powered products that now sit between search queries and the web. Unlike traditional search, where a human reads a ranked list of links and chooses one to click, AI answer engines synthesise responses from crawled content directly — if your content can't be read or is misunderstood, it won't be cited.
This differs from traditional SEO in important ways. Most AI crawlers don't execute JavaScript — they read raw HTML, just like early search engine crawlers did in the 1990s. A modern React SPA that renders beautifully in a browser may look like a blank page to GPTBot. AI systems also weight explicit, machine-readable structure (JSON-LD schema, llms.txt, well-formed headings) more heavily than link-based authority signals — because they're synthesising facts, not ranking documents.
AI readiness doesn't mean you should unilaterally open your site to all AI crawlers. For paywalled, licensed, or proprietary content, blocking AI training crawlers is a legitimate and commercially reasonable choice. The goal is to make conscious, informed decisions about each crawler — not to accidentally block them through outdated robots.txt rules written before AI crawlers existed.
The 6 things we check, and why
🤖 AI crawler access via robots.txt
robots.txt is the first thing an AI crawler reads before touching your site. If GPTBot, ClaudeBot, or PerplexityBot are disallowed, they won't crawl your pages regardless of how good your content is. This check shows which of the 14 major AI crawlers are allowed, blocked, or simply not mentioned. Not mentioned means open access — AI crawlers treat silence as permission, same as traditional web crawlers.
📄 llms.txt presence and quality
llms.txt is a new open standard (proposed in 2024 by fast.ai's Jeremy Howard) for giving AI systems a curated, structured summary of your site — similar to what robots.txt does for crawling rules, but positive and descriptive rather than restrictive. It's a Markdown file at /llms.txt that tells AI systems what your site is about, lists your most important pages, and provides context a crawler might miss. An optional /llms-full.txt can contain your full content corpus for systems that want more. Don't have one yet? <a href='/tools/llms-txt-generator' class='text-cyn hover:underline'>Generate one free →</a>
🏷️ Structured data (JSON-LD)
JSON-LD schema markup lets you embed machine-readable facts directly in your HTML — things like 'this page is an Article written by X on date Y' or 'this is a Product with these specs and this price'. AI systems and search engines both use this structured data to understand your content more reliably than parsing natural-language text alone. At minimum, Organization and WebSite schema help AI systems identify your brand. FAQPage and Article schema can get your content into rich results and AI answer snippets.
⚡ Content accessible without JavaScript
Most AI crawlers don't execute JavaScript — they fetch your HTML and read what's in it, the same way search engine crawlers have always worked. If your site is a JavaScript-only single-page app that renders content exclusively client-side, AI crawlers will see a near-empty page with a script tag and nothing else. Server-side rendering (SSR) or static-site generation (SSG) ensures your content exists in the raw HTML response, visible to any crawler regardless of JS execution.
🔖 Metadata and semantic markup
A clear <title>, a concise meta description, a single <h1>, Open Graph tags, and alt attributes on images all help AI systems parse and understand your pages. These are the basics of semantic HTML that have mattered for SEO for decades — they matter equally for AI readiness. A page without a title or with multiple competing <h1> tags is ambiguous to a crawler trying to understand what the page is about.
🗺️ Sitemap and crawling hygiene
A sitemap.xml at /sitemap.xml tells crawlers (AI and traditional) which pages exist on your site and when they were last updated. Without one, crawlers rely on link-following to discover pages, which means deep or orphaned pages may never get crawled. A robots.txt Sitemap: directive helps too — it tells every crawler where to find your sitemap without them having to guess.
AI crawler access doesn't have to be all-or-nothing
Blocking AI crawlers entirely is a legitimate choice for some site owners — paywalled content, licensed data, and proprietary research all have valid reasons to opt out. This tool's framing isn't "more AI access is always better." It's "are you making intentional choices, or are you accidentally open or accidentally blocked?" DataSec's own approach (see scraping protection) is about granular per-crawler policy — allowing Googlebot, auditing GPTBot, blocking unknown scrapers — rather than blanket rules in either direction.
Frequently asked questions
Want actual enforcement of your AI crawler policy, not just a score?
Sign up free — connect your site in minutes, no commitment, no CAPTCHA friction added during evaluation.