[Free tool]
Can AI crawlers reach your site?
We read your robots.txt and request your homepage as GPTBot, ClaudeBot, PerplexityBot and other AI crawlers. If they're shut out, the AI tools that rely on them can't read your pages.
[01]What you get/ sample result
1 of 6 AI crawlers are blocked
Read robots.txt and requested the homepage as each bot.
- GPTBotOpenAI · model trainingServer returned 200Allowed
- ChatGPT-UserOpenAI · fetches pages for usersServer returned 200Allowed
- OAI-SearchBotOpenAI · ChatGPT searchServer returned 200Allowed
- ClaudeBotAnthropic · web crawlerServer returned 200Allowed
- PerplexityBotPerplexity · search indexBlocked in robots.txtBlocked
- GooglebotGoogle · Search crawlerServer returned 200Allowed
From result to next step.
- 1
Run it on your domain
Type it in the box above. Results in seconds.
- 2
Copy the fix, if you need one
Any crawler that is blocked comes with the robots.txt lines to add. Then run the check again.
- 3
Track this every week
Free account, no card. The full scan re-checks your site weekly and shows whether AI names you, with every page behind each answer.
Scan free
[02]How it works/ and its limits
What this checks
- 1
Your robots.txt rules
We look for rules that shut each crawler out by name.
- 2
Your server's answer
We request your homepage as each bot. A 401, 403 or 451 means something at your server is turning it away.
- 3
The fix, if you need one
Anything blocked comes with the robots.txt lines to add, ready to copy.
Access comes before everything else.
AI answers built from live web search can only use pages their crawlers can fetch. A blocked crawler quietly cancels out your content, PR and link work, and it takes seconds to rule out.
What it doesn't tell you
This checks whether AI crawlers are excluded, not whether you're cited. Being allowed is a precondition for being cited, not a cause of it. A HEAD request can also differ from how a bot actually crawls, and some sites block by IP rather than user-agent — we can't see that from here.
[03]Questions/ about this tool
Good to know
Six: GPTBot, ChatGPT-User and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and Googlebot. For each one we read your robots.txt rules and request your homepage as that bot.
No. The check is free, needs no signup and only reads public files on your site.
Not by itself. Being allowed is a precondition for being cited, not a cause of it. The pages AI cites about your category matter more, and that is what the full scan shows.
Usually not. A 401, 403 or 451 tends to come from a firewall, CDN or hosting setting, so the block has to be lifted there.
[→] Free scan
Next: see whether AI actually names you.
The full scan asks the questions your buyers ask, answers them with AI and live web search, and shows every page each answer was built from. One domain, about 2 minutes, free account.
- Who AI names for your buying questions
- Every page each answer cited
- The pages to get listed on first
Want to look first? See how it works