AI Crawler Access: Which Bots to Allow and How

If you want ChatGPT, Perplexity, Claude, or Google's AI Overviews to read and cite your site, the first thing that has to be true is boring: the bots have to be able to fetch your pages. That sounds automatic, but in 2026 it often is not. Two separate gates can turn an AI crawler away: your robots.txt rules, and the bot-management layer in your CDN or web application firewall (WAF). This guide covers both, names the bots worth caring about, and shows the exact patterns to allow or block each one. It maps directly to two of AuditZap's nine AI Visibility checks: AI crawler access (robots.txt) and AI crawler firewall blocking (the network layer).

Which AI crawlers are we actually talking about?

There are a lot of user agents in the wild, but they fall into two buckets that matter for very different reasons. Training crawlers collect data to train models. Search and citation agents fetch pages live to answer a specific user's question, and they are the ones that decide whether you show up in an AI answer. Blocking the first group is a content-licensing decision with no effect on your AI search visibility. Blocking the second group quietly removes you from AI answers. AuditZap uses exactly this split when it grades your robots.txt.

One caveat on Google-Extended: it is a robots.txt control token rather than a crawler of its own - Googlebot does the fetching - and blocking it also opts your content out of Gemini model training. Google Search indexing and AI Overviews are governed by Googlebot, not Google-Extended.

The exact roster shifts as vendors rename agents, so treat any list as a snapshot. AuditZap keeps its list in one place and updates it against vendor docs.

Which AI crawlers should you allow?

For most sites the answer is simple: allow the search and citation agents, and make a deliberate choice about the training crawlers.

The search and citation agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, and Google-Extended) are the ones that fetch your pages to answer a live question. If you want to appear in AI answers, these must be allowed. Blocking them is almost always an accident, usually a copy-pasted "block all bots" rule that swept them up.

The training crawlers (GPTBot, ClaudeBot, CCBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, Amazonbot) collect data to train future models. Blocking them is a legitimate stance if you do not want your content used as training material, and it does not cost you any AI search visibility. Publishers who license their archives often block these while leaving the search agents open. That combination is perfectly coherent, and AuditZap treats a training-crawler block as an informational note, not a warning.

The one combination to avoid by accident is blocking the search agents while thinking you only blocked training. That is the mistake this check exists to catch.

What do the robots.txt patterns look like?

robots.txt uses per-user-agent groups. To block a single bot, name it and disallow everything under it:

User-agent: GPTBot
Disallow: /

That blocks OpenAI's training crawler and nothing else. To allow a specific bot everywhere, you generally do not need a rule at all: absent a matching group, most crawlers assume they are allowed. If you want to be explicit, an empty disallow allows everything:

User-agent: PerplexityBot
Disallow:

A realistic setup that blocks training but keeps you visible in AI answers looks like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Disallow:

User-agent: PerplexityBot
Disallow:

User-agent: Google-Extended
Disallow:

AuditZap parses your robots.txt into groups, separates the training bucket from the search bucket, and only warns when a search or citation agent is fully blocked with Disallow: /. A blanket User-agent: * with Disallow: / blocks everything, including the search agents, so that reads as a warning too.

How can a WAF or CDN block AI crawlers silently?

This is the part almost nobody checks, and it is where most accidental blocking happens now. Your robots.txt is an instruction that well-behaved crawlers choose to obey. Your CDN or WAF is a gate that enforces itself before a request ever reaches your page. The two are completely independent, and they can disagree.

Cloudflare and other providers now ship bot-management rules that challenge or block AI crawlers by default on many plans. When that is on, a request identifying as GPTBot receives an HTTP 403 (or 401, 429, or 503) while a normal browser loads the same page fine. Your robots.txt can say "yes, come in" and the crawler still bounces off the front door. Because the block never touches robots.txt, every robots-only checker misses it entirely.

You can reproduce it from the command line with two requests to your homepage: one with a normal browser user agent, one identifying as an AI bot. If the browser request succeeds and the bot request returns 401, 403, 429, or 503, your CDN is turning AI crawlers away. AuditZap runs exactly this differential probe on every audit: it fetches your page as a browser and again as GPTBot, and warns only when the browser succeeds but the bot is blocked. A block that hits both requests is not AI-specific (the site rejects non-browser fetches generally), so it does not warn on that. When it does find a mismatch, the fix is in your CDN's bot-management settings, not robots.txt. On Cloudflare that usually means allowing verified AI bots or adding an exception rule for the user agents above.

How do you check both gates at once?

Because the two gates are independent, checking one proves nothing about the other. A clean robots.txt with a WAF that 403s every bot is a common and invisible failure. The free llms.txt checker tests both together: it reads your robots.txt AI-crawler rules and runs the browser-versus-bot firewall probe in one pass, so you see whether either gate is closed. If you would rather generate a clean starting point for the file that tells assistants what matters on your site, the llms.txt generator builds a valid one from your pages. For the full picture, a free audit runs all nine AI Visibility checks and rolls them into a single score you can track over time.

FAQ

Does blocking GPTBot hurt my Google rankings?

No. GPTBot is OpenAI's training crawler and has nothing to do with Googlebot or your classic search rankings. Blocking it only opts your content out of being used as OpenAI training data. Your position in Google's blue links is unaffected.

If I block training crawlers, do I disappear from ChatGPT answers?

No, as long as you leave the search and citation agents allowed. ChatGPT's live search uses OAI-SearchBot and ChatGPT-User, which are separate from GPTBot. Block GPTBot for training if you like, but keep the search agents open and you stay eligible to be cited.

Why does my site pass robots.txt but still not get crawled by AI bots?

Almost always a CDN or WAF bot rule. robots.txt is advisory and the crawler obeyed it, but your firewall blocked the request at the network layer before it mattered. Run the browser-versus-bot probe: if the bot request returns 403 or 503 while a browser succeeds, fix it in your CDN bot-management settings.

Is there a downside to allowing all AI crawlers?

The main considerations are server load and whether you want your content used for model training. Allowing the search agents has no real downside if you want AI visibility. Allowing training crawlers is a values-and-licensing question, not a technical risk for most sites.

How often should I re-check crawler access?

Any time you change CDN settings, migrate hosts, or turn on a new security feature, because those are exactly the moments a bot rule gets flipped on without you noticing. Otherwise a monthly check is plenty. The llms.txt checker makes it a ten-second job.