AI Crawler Checker
Check robots permission and real server responses for GPTBot, ClaudeBot, PerplexityBot and more.
What does this tool check?
Four gates decide whether AI can read your site: robots.txt permission, whether the server actually lets the crawler UA through (WAFs often 403 GPTBot even when robots.txt allows it), how much content survives without JavaScript, and meta/header blocking directives.
How should I read the result?
We probe from our own servers, so we can only test UA-level blocking. Every probe ships with a normal-browser control request: if the control itself fails, verdicts show "inconclusive" — never a false "blocked". Google-Extended and Applebot-Extended are robots tokens only, so they are marked not probeable.
What should I do next?
If a crawler is blocked at the server level, fix it in your CDN/WAF bot settings — robots.txt cannot help there.
Sources: OpenAI crawler docs · RFC 9309 Robots Exclusion Protocol
Access rules change without warning
robots.txt edits and CDN bot rules usually ship without anyone telling you. InsightWonder re-checks on a schedule and alerts you when a crawler starts getting blocked.
Run a full analysis- Scheduled re-checks with change alerts
- Visibility and citations across five AI engines
- Fixes ranked by impact, with content generated