[Free tool]
robots.txt rules for AI crawlers
We read your live robots.txt, find the AI crawlers it shuts out, and write the exact lines to add. Built from what's blocking you today, not a generic template.
[01]What you get/ sample result
# Allow PerplexityBot to crawl your siteUser-agent: PerplexityBotAllow: / # Allow GPTBot to crawl your siteUser-agent: GPTBotAllow: /From result to next step.
- 1
Run it on your domain
Type it in the box above. Results in seconds.
- 2
Paste the Allow lines
One small group per blocked crawler, ready to add to your robots.txt. Review it first if you block bots on purpose.
- 3
Track this every week
Free account, no card. The full scan re-checks your site weekly and shows whether AI names you, with every page behind each answer.
Scan free
[02]How it works/ and its limits
What this checks
- 1
Read your live file
We fetch robots.txt from your domain and look for rules that disallow each AI crawler.
- 2
Test each crawler
We request your homepage as each bot, to catch blocks that happen at your server instead.
- 3
Write the Allow lines
You get one small group per blocked crawler, ready to paste.
One old line can hide your whole site.
robots.txt is the first file a crawler reads. A single Disallow rule, sometimes added years ago by a plugin or a host, can keep AI search crawlers away from every page you publish.
What it doesn't tell you
This generates rules for the crawlers we found blocked. It does not rewrite your existing robots.txt — review before pasting, especially if you block crawlers deliberately. Allowing a crawler does not mean it will visit, and visiting does not mean you'll be cited.
[03]Questions/ about this tool
Good to know
No. It writes only the lines to add for the crawlers we found blocked. Your existing rules stay as they are.
Then change that group's Disallow line instead of adding a second group for the same crawler. Run the check again afterwards.
Keep those rules. Only add the lines for crawlers you want to let in.
Not necessarily. Allowing a crawler does not mean it will visit, and a visit does not mean you will be cited.
[→] Free scan
Rules fixed? See what AI says about you.
The full scan asks the questions your buyers ask, answers them with AI and live web search, and shows every page each answer was built from. One domain, about 2 minutes, free account.
- Who AI names for your buying questions
- Every page each answer cited
- The pages to get listed on first
Want to look first? See how it works