Assistant Crawlers: Who to Allow and How to Verify

Learn which assistant crawlers to allow for AI citations, how to verify Googlebot and GPTBot by IP, and how to block unwanted bots with robots.txt.

ADS Beast editorial teamPublished Updated 12 min read

This article was prepared with AI. The ADS Beast editorial team is responsible for its content.

Assistant crawlers are automated bots that fetch your pages on behalf of AI systems such as ChatGPT, Claude, Perplexity and Google Gemini. You decide who gets in through robots.txt, then verify identity at the server level. Allowing the right ones keeps you citable in AI answers; blocking the wrong ones removes you from them.

Short version

  • Assistant crawlers fall into two groups: retrieval bots that fetch pages to answer a live question, and training bots that collect text for model development. Blocking a retrieval bot removes you from that assistant's answers.
  • GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and Google-Extended are the tokens most sites need to make a decision about.
  • Google-Extended is separate from Googlebot. Blocking it does not remove you from Google Search.
  • User-agent strings prove nothing. Verify by IP range and reverse DNS, then forward-confirm.
  • robots.txt is a request, not enforcement. Hard blocking happens at the server, WAF or CDN.

Which assistant crawlers matter for your site?

The tokens most sites start with are GPTBot, ClaudeBot, PerplexityBot and Google-Extended, but they do not all do the same job. GPTBot and ClaudeBot collect content that may be used for model training, PerplexityBot surfaces and links sites in Perplexity's search results, and Google-Extended controls Gemini training and grounding. OpenAI and Anthropic also run separate search and user-request agents: OAI-SearchBot and ChatGPT-User, Claude-SearchBot and Claude-User.

The distinction between retrieval and training is the one that costs people traffic. A retrieval bot fetches your page at the moment a user asks a question, so the assistant can quote or link you in the answer. A training bot collects your text into a dataset for later model versions. If you block a retrieval bot, you disappear from that assistant's answers today. If you block a training bot, you lose influence over future model behavior but keep your current citations.

Each vendor documents which agent does what (OpenAI, Anthropic, Perplexity), so check the documentation before you decide. The token name tells you the vendor, not the purpose.

Crawler tokenVendorWhat it typically doesBlocking it affects
GPTBotOpenAICrawls content that may be used to train OpenAI modelsModel training only; ChatGPT search is controlled by OAI-SearchBot
ClaudeBotAnthropicCollects content that may be used to train Claude modelsModel training; Claude search and user requests use Claude-SearchBot and Claude-User
PerplexityBotPerplexitySurfaces and links sites in Perplexity search resultsYour presence in Perplexity search results
Google-ExtendedGoogleControls Gemini training and grounding in Gemini Apps and Vertex AIGemini only; Search and AI Overviews follow Googlebot
GooglebotGoogleSearch indexing and rankingYour Google Search rankings

The table is a starting map, not a contract. Vendor behavior changes, and the only reliable source is the vendor's own crawler documentation, which you should re-read whenever you make a policy decision.

Retrieval bots vs training bots. Retrieval bot: Fetches pages live to answer a user question, drives citations; Training bot: Collects text for future model versions, no live citation; Block retrieval: You disappear from that assistant's answers today; Block training: You keep current citations, los
Blocking the wrong group costs citations you cannot easily win back.

Googlebot and Google-Extended are not the same thing

Googlebot fetches pages so Google can index and rank them in Search. Google-Extended is a robots.txt token that controls whether your content is used to train Gemini models and to ground answers in Gemini Apps and Vertex AI. Blocking Google-Extended leaves your Search rankings untouched.

This trips up more site owners than any other point on the list. People see "Google" in both names, assume one switch controls everything, and either block Googlebot by accident or leave Google-Extended open when they meant to close it.

Googlebot obeys robots.txt for crawling. It does not use robots.txt to decide indexing, which you control with meta tags and HTTP headers instead. So a Disallow rule for Googlebot limits what Google can fetch, while a noindex directive limits what Google can show. They do different jobs.

Google-Extended works the other way around. It is a permission token with no crawling role of its own. You either grant or withhold permission for Gemini training and grounding, and nothing about your ordinary Search presence changes either way.

If your goal is Search visibility plus AI answer visibility, you leave both open. If your goal is Search visibility without feeding Gemini, you keep Googlebot open and block Google-Extended. Those are two separate decisions and you should make them separately. AI Overviews and AI Mode are part of Search, so Googlebot rules, not Google-Extended, decide whether your pages can appear there (Google's guidance on AI features).

How to verify a crawler is really from Google

Verify Googlebot by matching the source IP against Google's published IP ranges, then running a reverse DNS lookup that resolves to a hostname ending in googlebot.com or google.com, then a forward lookup on that hostname that returns the same IP. Any step that fails means the request is spoofed.

The check has several steps because each one alone is forgeable or incomplete:

  1. Get the source IP of the request from your server logs.
  2. Match it against the official IP ranges Google publishes for its crawlers.
  3. Run a reverse DNS lookup on the IP. The hostname must end in googlebot.com or google.com.
  4. Run a forward DNS lookup on that hostname. It must resolve back to the same IP you started with.
  5. If every check passes, treat the request as genuine Googlebot. If any step fails, treat it as a fake.

The forward-confirm step is the one people skip. Reverse DNS alone can be faked by anyone who controls the PTR record for their own IP block, which is why the round trip back to the original IP is the part that actually proves anything.

User-agent strings prove nothing on their own. Anyone can send a request with "Googlebot" in the header. Log analysis tools that filter by user-agent alone will show you fake Googlebot traffic as if it were real, and you will make policy decisions on bad data.

The same logic applies to other vendors. OpenAI, Anthropic and Perplexity publish JSON lists of their crawler IP addresses in their bot documentation, so for them you match the source IP against the published list. If a vendor does not publish ranges, you have less to verify against, and you should treat unverifiable traffic with more caution rather than less.

Verify a crawler in four checks. Get the source IP: Pull it from your server logs for the request in question; Match against published ranges: Compare the IP to the vendor's official crawler IP list; Reverse DNS lookup: Hostname must end in googlebot.com or google.com; Forward-confirm the hostname:
Any failed step means the request is spoofed, not genuine.

How to block an unwanted AI crawler with robots.txt

Add a user-agent line with the bot's exact token, then a Disallow rule under it. Its position in the file does not matter: a crawler follows the group with the most specific matching user agent, and Google states that the order of groups is irrelevant (robots.txt specification).

A working block looks like this:

User-agent: GPTBot
Disallow: /

Two things go wrong here often. First, people use the wrong token. "GPT" is not a token. "OpenAI" is not a token. The token must match exactly what the vendor publishes, or the rule is ignored. Second, people expect a bot that has its own group to obey the wildcard group as well. It does not: a crawler follows only the most specific group that matches it, so repeat in the specific group every rule you want that bot to follow.

Robots.txt is voluntary. It is a convention that well-behaved crawlers follow and bad actors ignore. Treat every rule in it as a request.

If you also want AI assistants to understand your site structure and content priorities, publishing an llms.txt file for assistant crawlers gives cooperative systems a map they can actually use. It is not a substitute for robots.txt, and it is just as voluntary.

Why some crawlers ignore your robots.txt

Robots.txt is a convention, not a security control. Scrapers, malicious bots and some unlisted crawlers skip it entirely, and no text file can stop them.

For those, enforcement moves to the server layer:

  • Rate limiting per IP and per user-agent, so a scraper cannot pull your whole site in minutes.
  • WAF rules that return 403 to suspicious user-agent strings and known-bad IP ranges.
  • IP allowlisting for known good bots, which inverts the model: everything is blocked unless it proves it belongs.
  • CDN-level bot management, which filters before requests reach your origin.

Allowlisting is the strongest option and the most maintenance. You get hard enforcement, but you take on the job of keeping the list current as vendors add and retire IP ranges. For most sites, a combination of rate limiting and WAF rules catches the bulk of abuse without the upkeep.

The tradeoff is real. Aggressive blocking catches legitimate crawlers too, and a false positive on a retrieval bot quietly removes you from an assistant's answers without any error message anywhere. If AI citations are part of your traffic strategy, review your block rules after every change and check whether the assistants still mention you. An audit of what AI answers say about your company shows you the effect of a block sooner than a ranking report would.

Verification at scale: logs, sampling and false positives

At small volume you can verify crawlers by hand. Past a few thousand requests a day, you need a process, because manual checks stop happening and stale assumptions take over.

A workable process has three parts. First, log the source IP and user-agent for every request that claims to be a known crawler. Second, run the reverse-forward DNS check automatically on a sample, not on every request, and alert when the failure rate moves. Third, keep a short list of confirmed vendor IP ranges and refresh it on a schedule, since vendors add ranges without announcing it to you.

The failure mode to watch for is a false positive that nobody notices. You block an IP range because it looked suspicious, a retrieval bot was inside it, and your citations in that assistant drop to zero. Nothing in your analytics flags this, because AI answer traffic often arrives without a referrer you can attribute. This is the same attribution problem that shows up in ad measurement when the same period shows different numbers: the signal exists, but it is not where you expect to find it.

If AI-driven traffic feeds into your conversion tracking, make sure server-side events are set up so you can enable server-side conversion tracking without double-counting. Otherwise you will see the traffic in one system and the conversions in another, and you will not be able to tell whether a crawler policy change helped or hurt.

What to allow, in practice

Allow retrieval bots you want citations from. Allow training bots if you want influence over future model behavior and have no licensing objection. Block everything you cannot verify. That order of operations keeps you citable without opening the door to scrapers.

A concrete starting policy for most content sites:

  • If AI citations are part of your strategy, allow the search and user-request agents: OAI-SearchBot and ChatGPT-User, Claude-SearchBot and Claude-User, PerplexityBot. Decide on the training crawlers GPTBot and ClaudeBot separately.
  • Decide on Google-Extended separately from Googlebot, based on whether you want Gemini training and grounding.
  • Block unverifiable traffic at the server, not in robots.txt.
  • Re-check your rules quarterly against vendor documentation, because tokens and behaviors change.

The right answer depends on your business model. A publisher with licensing revenue has a different calculus than a SaaS company that wants to be mentioned in every relevant AI answer. Neither is wrong. What is wrong is leaving the decision unmade and letting defaults decide for you.

When AI-sourced traffic becomes a meaningful share of conversions, the crawler policy stops being a technical detail and becomes a channel decision. That is the point where measurement discipline matters, and where teams that treat AI answers as a real acquisition channel start to separate from those that treat it as noise; ADS Beast keeps a history of brand mentions in AI answers for that purpose.

Next step

Open your server logs and pull every request from the past seven days that claims to be GPTBot, ClaudeBot, PerplexityBot or Googlebot. Check a sample of each against the vendor's published IP list, and run the reverse-forward DNS check for Googlebot. Whatever fails verification, block at the server. Whatever passes, confirm you actually want it, then leave it alone.

See how this works in ADS Beast.