AI Bot Policy Builder
Build explicit robots.txt crawler policies from your content access choices.
Allow or block, per purpose
Blocking training bots does not block search-visibility bots — different user agents. Keep the second group if you want AI search visibility.
# AI crawler policy (generated by InsightWonder Tools) # Model training: allow User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Bytespider User-agent: Meta-ExternalAgent User-agent: CCBot Disallow: # AI search visibility: allow User-agent: OAI-SearchBot User-agent: PerplexityBot Disallow: # On-demand user fetch / assistants: allow User-agent: ChatGPT-User User-agent: Amazonbot Disallow:
What does this tool check?
Pick, per AI operator and purpose (training, search, user-fetch), whether to allow or block — and get a ready-to-paste robots.txt block with the correct real user-agent tokens.
How should I read the result?
Blocking training bots (GPTBot, ClaudeBot) does not block search visibility bots (OAI-SearchBot, PerplexityBot) — they are different user agents with different purposes. The generator keeps these separate on purpose.
What should I do next?
Paste the block into your robots.txt, then verify with the AI Crawler Checker that servers actually honor it.
Sources: RFC 9309 Robots Exclusion Protocol · OpenAI crawler docs
Access rules change without warning
robots.txt edits and CDN bot rules usually ship without anyone telling you. InsightWonder re-checks on a schedule and alerts you when a crawler starts getting blocked.
Run a full analysis- Scheduled re-checks with change alerts
- Visibility and citations across five AI engines
- Fixes ranked by impact, with content generated