There is no single right answer to this; it depends on what you want from AI search. The mechanism, though, is simple and worth getting straight first. You control AI crawlers the same way you control search crawlers, through your robots.txt file, where each crawler is named and either allowed or disallowed. Block them if your goal is to keep your content out of AI training datasets and out of AI generated answers. Allow them if your goal is to be cited and surfaced in AI search, because a crawler that cannot read you cannot quote you. The decision is not all or nothing, since each system uses its own named crawler, so you can allow the ones that send you visibility and block the ones that only take. WPBuildAI scans your site and returns your URL list and structure, so you can see exactly what a crawler would reach before you decide what to open or close.
The trade is lopsided enough to be worth seeing in numbers: Cloudflare measured Anthropic crawling about 38,065 pages per referred visit and OpenAI about 1,091, against roughly 5.4 for Google, so blocking a training crawler forfeits very little traffic while blocking Google forfeits a great deal.
How crawler control actually works
Before deciding, understand the tool. A robots.txt file at the root of your site lists user agents, the names crawlers identify themselves with, and gives each a set of allow and disallow rules. This is the same file that governs search engines, explained in the introduction to robots.txt. Crawlers that respect the standard read this file before they read your pages and obey the rules that match their name. That last detail is the key to everything below: control is per crawler, by name. You are not flipping one switch for all of AI; you are writing a short list of who may read what.
The case for allowing them
If you want to appear in AI answers, you have to let the relevant crawlers in. AI search tools can only summarize, cite, and link to content they were able to read. Block the crawler and you remove yourself from that surface entirely, the way a blocked search crawler removes you from search. For most businesses that publish to be found, being readable is the point, and AI search is simply a newer place to be found. Allowing these crawlers is the equivalent of keeping your front door open during business hours. It does not guarantee visitors, but closing it guarantees none.
The case for blocking them
There are honest reasons to block, too. You may not want your writing used to train models, or you may sell access to content that loses value if it is freely summarized. Operators publish their crawler names so you can target them; OpenAI documents GPTBot and its other agents in the OpenAI bots reference, and similar names exist for other systems. If your content is your product rather than your advertisement, blocking the crawlers that train on it can be the right call. The trade is explicit: you trade away presence in AI answers to keep control of how your content is used.
You can allow some and block others
The most useful insight is that this is not a binary. Because every system uses its own named crawler, listed in references like the Google crawlers overview, you can write a mixed policy. Allow the crawlers tied to AI search products that link back and send you readers, and block the crawlers that only collect training data and return nothing. This per crawler approach lets you pursue AI visibility while declining uses you dislike. It takes a few more lines than a blanket rule, but it matches your policy to your actual goals instead of forcing one extreme on you.
What blocking does not do
A blocked crawler is a rule about the future, not the past. Disallowing GPTBot today stops that crawler from reading you going forward, but it does not erase content a model already learned, and it does not purge pages other tools cached earlier. If your aim is removal of something already out there, robots.txt is the wrong instrument; it never reaches back. Be clear eyed about this when you decide, because the relief of adding a disallow line can create a false sense that existing exposure has been undone. It has not. Blocking shapes what happens from now on.
The real risk: blocking too much by accident
The most common way this goes wrong has nothing to do with AI. A rule meant to block one crawler, written with a careless wildcard or placed under the wrong user agent, ends up disallowing everything, including search engines. The site quietly falls out of search, and weeks pass before anyone connects the drop to a one line edit. Because AI crawlers and search crawlers have different names, a precise rule is safe, but an imprecise one is dangerous. Always confirm your rule targets only the crawler you intend, and that your important pages remain allowed for search.
Steps to set your crawler policy
- Decide your goal: presence in AI answers, control over training, or a mix.
- List the crawler names you want to allow and the ones you want to block.
- Write per crawler rules in robots.txt, one user agent block at a time.
- Leave search crawlers allowed unless you have a specific reason not to.
- Test the file so no rule accidentally disallows your whole site.
- Recheck after a migration, since a new platform may ship its own robots.txt.
Worked example: a mixed policy
Imagine a publisher that wants readers from AI search but does not want its archive used purely for training. The team starts by scanning the site to see every URL a crawler could reach. They then write a robots.txt that allows the crawlers tied to AI search tools that cite and link sources, so their articles can appear in answers with attribution. Under separate user agent blocks, they disallow the crawlers that only gather training data. They leave all search crawlers allowed. Finally they test the file to confirm nothing blocks search by accident. The result is a policy that pursues visibility where it pays and declines use where it does not, expressed in a handful of precise lines rather than one blunt block.
Limitation: respect is voluntary and policies change
It is fair to name the limits. The robots.txt standard relies on crawlers choosing to obey it, so a well behaved crawler follows your rules while a badly behaved one may ignore them. Robots rules are the right tool for compliant crawlers, not a hard wall against every actor. Crawler names and operator policies also change over time, so a policy written once will drift, and a name you blocked may be retired or renamed. Treat your crawler policy as something you maintain, not something you set once. For the broader picture of how AI systems read and index sites, see how generative AI indexes WordPress sites.
Common mistakes
- Assuming there is one correct answer rather than a decision about goals.
- Writing a blanket block that disallows search engines along with AI crawlers.
- Believing a disallow rule removes content models already learned.
- Blocking every AI crawler when some send cited, linked traffic.
- Forgetting that a migration can replace your robots.txt with a default.
Key points
Whether to block AI crawlers like GPTBot depends on your goal, not on a universal rule. Block them in robots.txt to keep content out of AI training and answers; allow them to appear in AI search, since a crawler that cannot read you cannot cite you. Because each system uses its own named crawler, you can allow the ones that send linked traffic and block the ones that only take. Remember that blocking shapes the future and does not erase what was already learned, and that a careless rule can disallow search engines and cost real traffic, so test before publishing. If you want AI tools to find your best pages once you have allowed them, see do you need an llms.txt file and the generative engine optimization guide. WPBuildAI scans your site and returns the URL list and structure, so you can see what a crawler would reach before you decide. Send your web address for a free analysis.
Not affiliated with WordPress, Lovable, Webflow, Shopify, Wix, or Squarespace.