Scans the one page you enter, plus robots.txt, llms.txt and the sitemap on that site. Public pages only.

› Checking robots.txt, testing crawler access, reading the page…

What gets checked

  • robots.txt, evaluated per bot with real matching rules (longest match, Allow beats Disallow on ties) for 15 AI crawlers
  • Server blocking, whether your firewall or CDN refuses AI crawler user agents that a normal browser gets through
  • Raw-HTML content, because most AI crawlers do not run JavaScript
  • Structured data, JSON-LD presence and whether it parses
  • llms.txt, ai.txt, sitemap, plus noindex, title, description, H1, canonical and response time

Honest limits: server tests send crawler user agents from our servers, so they can't reproduce IP-verified allow-lists. A "refused" result on a site that verifies bot IPs is expected, and real bots still get through.

Frequently asked questions

What's the difference between search bots and training bots?

Search and browsing bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot) fetch pages so assistants can cite them. Training crawlers (GPTBot, ClaudeBot, Google-Extended) collect content for model training. Blocking training crawlers doesn't remove you from AI answers; blocking search bots does.

Why was a bot "refused" when real bots are fine?

We send real crawler user agent strings from our own servers. Sites that verify crawler IP addresses correctly refuse that. If only some bots are refused and you verify IPs on purpose, you're fine. If every bot is refused, look at your firewall or CDN bot settings.

Does JavaScript-rendered content matter?

Yes. Most AI crawlers read raw HTML only. If your text appears only after scripts run, they see an almost empty page.

Is llms.txt required?

No. It's an emerging convention with uneven adoption. It's cheap to add and we report it, but it's a small part of the score and not a proven ranking factor.

Does the checker store my data?

Scan results are not stored. To prevent abuse we keep a hashed requester address and the scanned hostname for rate limiting, deleted automatically after a day.