robots.txt Checker
Parse allow and disallow rules per crawler and highlight conflicts.
What does this tool check?
We fetch /robots.txt, parse every User-agent group with the standard longest-match rule, and show which groups block the site root, which sitemaps are declared, and which lines no parser understands.
How should I read the result?
A missing robots.txt means "allow everything" — that is a valid state, not an error. robots.txt controls crawl permission; it does not guarantee indexing or citation.
What should I do next?
If you intend to allow AI crawlers, make sure no User-agent group accidentally matches them with a Disallow: /.
Sources: RFC 9309 Robots Exclusion Protocol
Access rules change without warning
robots.txt edits and CDN bot rules usually ship without anyone telling you. InsightWonder re-checks on a schedule and alerts you when a crawler starts getting blocked.
Run a full analysis- Scheduled re-checks with change alerts
- Visibility and citations across five AI engines
- Fixes ranked by impact, with content generated