01Which AI crawlers exist and what each does
The big AI companies now publish separate user agents for separate jobs, which makes the choice less all-or-nothing than it was in 2023.
| User agent | Company | Used for | If you block it |
|---|---|---|---|
| GPTBot | OpenAI | Training foundation models | Content not used for training |
| OAI-SearchBot | OpenAI | ChatGPT search results | Not shown in ChatGPT search answers |
| ChatGPT-User | OpenAI | Fetching a page a user asked about | robots.txt may not apply (user-initiated) |
| ClaudeBot | Anthropic | Training | Content excluded from training |
| Claude-SearchBot | Anthropic | Indexing for Claude's search | Less visible in Claude search results |
| Claude-User | Anthropic | Fetching for a user's question | Claude cannot fetch your pages for users |
| Google-Extended | Gemini training and grounding | No effect on Google Search ranking |
02What the companies say
OpenAI's crawler documentation says disallowing GPTBot indicates content should not be used to train its models, that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, and that robots.txt changes take about 24 hours to apply. It notes that ChatGPT-User visits are initiated by users, so robots.txt rules may not apply.
Anthropic's help article on its crawlers describes ClaudeBot for training, Claude-User for fetching pages in response to users and Claude-SearchBot for search, all respecting robots.txt. Google's crawler documentation says Google-Extended controls use for Gemini training and grounding and does not affect inclusion or ranking in Google Search. AI Overviews are part of Search and use Googlebot.
03The trade-off
Blocking AI crawlers is a reasonable choice for publishers whose content is the product: paywalled journalism, original research, illustrators and writers who do not want their work used for training. For them, the value of the content outweighs referral traffic.
For most businesses the reverse is true. People increasingly ask assistants which tool, service or shop to use. If the assistant cannot read your site, it describes your competitors instead. Pew Research found that Google users who saw an AI summary clicked a traditional result in 8% of visits versus 15% without one, which cuts both ways: fewer clicks overall, but being the cited source is where the remaining attention goes. How to show up in Google AI Overviews covers that.
04robots.txt rules for each choice
Pick the one that fits. Rules go in /robots.txt at your domain root.
- 1Allow everything (most businesses)No AI-specific rules needed. Make sure no blanket Disallow: / applies to all user agents.
- 2Allow AI search, block trainingUser-agent: GPTBot, then Disallow: /. Repeat for ClaudeBot and Google-Extended. Leave OAI-SearchBot, Claude-SearchBot and Googlebot allowed.
- 3Block all AI useDisallow GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Claude-User and Google-Extended. Expect to disappear from AI answers.
- 4Check it workedLoad yoursite.com/robots.txt in a browser and watch your server logs for the user agents over the next few days.
05Server load and badly behaved bots
Some site owners block AI crawlers because aggressive crawling overloads small servers. Many crawlers honor Crawl-delay or rate limits, and a CDN or firewall can throttle unknown bots without blocking the major, well-behaved ones. robots.txt is a request, not a lock: bots that ignore it need blocking at the server or CDN level.
If you are unsure whether AI tools read your site well, does llms.txt work? and how to make a website AI friendly cover the rest of the setup.
What people who tried it say
Hacker News is split: many developers block AI crawlers over load and consent, while others note that blocking after models were trained changes little and that search visibility now depends on these bots.
- GPTBot – OpenAI's Web CrawlerHacker News
Commenters welcomed a standard robots.txt opt-out but pointed out it does not affect content already used for training.
- Devs say AI crawlers dominate traffic, forcing blocks on entire countriesHacker News
Operators of small sites describe AI crawler traffic overwhelming servers and debate tarpits, proof-of-work walls and blocks.
Looking for more first-hand opinions? Search Reddit for “should i block ai crawlers”. We link the live threads rather than summarizing them, so you read them in full.
Our take
if your website exists to win customers, allow AI search crawlers. Blocking training bots is a fair personal choice, but blocking search bots mostly hands the answer to a competitor.
Where this falls short
Crawler names and policies change often, and companies decide which bots honor which rules. Check each company's current documentation before relying on a rule, and remember robots.txt does not stop bad actors.
Sites published with Orlume allow search and AI crawlers by default, so assistants can read and cite your pages. The free plan includes 10 message credits.
Free to start · No card required · Live in one click
Sources
- 1.Overview of OpenAI crawlers · OpenAI
- 2.Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic
- 3.Google's common crawlers · Google Search Central
- 4.Google users are less likely to click on links when an AI summary appears · Pew Research Center
Researched and written by the Orlume team · Updated October 2026
Still stuck?