Crawler access · check 1 of 5 in the AI Visibility Audit
Can AI crawlers actually read your site?
The fast, focused check: enter your URL and see which AI bots your site allows or blocks — from robots.txt and a live user-agent test. Want the whole picture instead? Run the full audit.
No signup, no crawl of your whole site — we look at your homepage and robots.txt, the way an AI crawler first would.
We fetch your homepage as GPTBot, ClaudeBot, PerplexityBot and more — plus a normal browser as a control — and read your robots.txt directives verbatim.
A clear allow/block matrix per crawler, grouped by what they do. Honest by design: if your WAF challenges everything, we say the test is inconclusive rather than cry wolf.
How is this different from the AI Visibility Audit?+
This is the audit's first check, on its own. It answers one question fast — can AI crawlers reach your site? The full audit runs four more: what AI actually reads on your pages, what your platform ships by default, what frontier models already know about your brand cold, and how off-site sources corroborate you. If access is all you need right now, this is quicker; if you want the whole picture with every finding linked to evidence, run the audit.
Which crawlers do you test?+
The AI answer/search crawlers that matter most — OAI-SearchBot, PerplexityBot, Claude-SearchBot — plus user-triggered fetchers (ChatGPT-User, Perplexity-User, Claude-User), training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider), and Googlebot/Bingbot as a baseline.
What's the difference between the robots.txt result and the user-agent test?+
robots.txt is a stated policy we can read and verify directly — it's the ground truth for what you've asked crawlers not to fetch. The user-agent test is a live fetch as each bot to catch server- or WAF-level blocks that robots.txt won't show. Together they cover both what you declared and what your server actually does.
Why does it sometimes say “inconclusive”?+
If a plain browser is itself challenged by your bot-protection, we can't tell a real block from a WAF challenge — so we report inconclusive instead of guessing. Many WAFs allow-list real AI crawlers by verified IP even when they challenge a user-agent test, so a challenge here isn't proof the crawler is blocked.
Does this test whether the crawler actually obeys robots.txt?+
No — that's each vendor's published commitment, not something a probe can verify. We read what your robots.txt says and what your server returns; we don't claim to test a crawler's honesty.
Is anything stored?+
No. We check your public homepage and robots.txt in the moment and return the result to your browser — no account, no saved report, nothing retained. (You can opt a domain out of public checks entirely via /audit/opt-out.)
This is one check of five.
You just ran the crawler-access check. The full audit adds the other four — what AI reads on your pages, your platform defaults, what models already know about your brand, and off-site signals — all linked to the evidence.