AI crawler access
Whether AI crawlers — GPTBot, ClaudeBot, PerplexityBot and others — are actually allowed to fetch your pages. They are separate from Google, and they are frequently blocked without anyone deciding to block them.
Why it matters
If they are blocked you are invisible inside ChatGPT, Claude and Perplexity, and nothing in Google Analytics will tell you, because a blocked crawler leaves no trace in your traffic reports.
How to fix it, step by step
Moderate — an hour or so, or a plugin setting plus a re-check.
- 1Check robots.txt for any "Disallow" under user-agents named GPTBot, ClaudeBot, CCBot, PerplexityBot or Google-Extended.
- 2Check your CDN or firewall separately — this is the usual cause. Cloudflare has a "Block AI Scrapers and Crawlers" toggle that is ON by default on some plans, and it overrides anything robots.txt says.
- 3In Cloudflare: Security → Bots, and review the AI scraper setting.
- 4Decide deliberately. Allowing them means being citable in AI answers; blocking them means protecting content from training. Both are valid — the failure is not choosing.
How to know it worked
Re-run the CRO9 scan. The AI access category should report the crawlers as reachable rather than blocked, and the report will name which party refused if any still are.
Others in AI Access
Want to know which of these apply to your site?
The free scan runs all 33 checks against your pages and tells you which are failing, with the answer beside each one. No signup for the check itself.
Scan my site free