How to block AI scrapers in WordPress

AI companies send crawlers across the web for two reasons: to collect content for training their models, and to find pages their chatbots can cite when they answer questions. By default, WordPress lets all of them in. Most of us are perfectly happy to allow the crawlers that lead to our sites getting more visitors. Some of us would rather our content not be used to train their models.

In this tutorial, we’ll take a look at which AI crawlers are out there, how to block the ones you don’t want in your site’s robots.txt file, and what to do on WordPress.com, where there’s a built-in setting for it.

Training crawlers and search crawlers

Most AI companies use separate crawlers for training and for search, and you can block them one by one. That matters, because blocking a training crawler keeps your content out of future models, while blocking a search crawler keeps your site out of the answers, and the links, that chatbots show today.

A few of these need an explanation:

• Google-Extended and Applebot-Extended aren’t crawlers of their own. They’re names you can block in robots.txt to keep the content that Googlebot and Applebot collect out of AI training, without leaving Google or Apple’s search results. Google’s AI Overviews use Googlebot, though, so you can’t opt out of those without opting out of Google Search.
• Common Crawl isn’t an AI company, but its archive of the web is one of the most common sources of AI training data.
• The user crawlers fetch a page when someone asks a chatbot to read it. Since a person asked for it, some of the user crawlers don’t follow robots.txt at all.

A good middle ground is to block the training crawlers and keep the search crawlers, so that your content isn’t used for training, but chatbots can still find and link to it. That’s what we’ll do in this tutorial.

For what it’s worth, I don’t block any AI crawlers on this site. My tutorials answer the kind of questions people ask chatbots, and I’d rather be the source they cite in Google’s AI Overviews and ChatGPT than be left out of the answer. If I wrote fiction, sold illustrations or ran a recipe blog, I’d probably feel differently.

How to block AI training crawlers in robots.txt

The robots.txt file tells crawlers which parts of a site they’re allowed to visit. WordPress doesn’t have a robots.txt file by default, but generates one on the fly whenever it’s requested, and you can add your own rules to it with the robots_txt filter. For other ways to change it, see how to edit robots.txt in WordPress.

The best place for the filter is a must-use plugin, so it keeps working whichever theme you use, and can’t be deactivated by accident:

1. Connect to your site with an SFTP client, and open the wp-content folder.
2. Create a folder called mu-plugins, if there isn’t one already.
3. In it, create a file called block-ai-crawlers.php with the code below.

<?php
/**
 * Plugin Name: Block AI Training Crawlers
 * Description: Asks AI training crawlers not to crawl the site.
 */
/**
 * Adds AI training crawlers to robots.txt.
 *
 * @param string $output    The robots.txt output.
 * @param bool   $is_public Whether the site is public.
 * @return string The modified robots.txt output.
 */
function my_block_ai_crawlers( $output, $is_public ) {
	if ( ! $is_public ) {
		return $output;
	}
	$crawlers = array(
		'GPTBot',
		'ClaudeBot',
		'Google-Extended',
		'Applebot-Extended',
		'Meta-ExternalAgent',
		'Amazonbot',
		'Bytespider',
		'CCBot',
	);
	$output .= "\n";
	foreach ( $crawlers as $crawler ) {
		$output .= 'User-agent: ' . $crawler . "\n";
	}
	$output .= "Disallow: /\n";
	return $output;
}
add_filter( 'robots_txt', 'my_block_ai_crawlers', 99, 2 );

This adds a group of rules at the end of robots.txt that tells the training crawlers in the list not to visit any page on the site. Search engines and the AI search crawlers aren’t in the list, so they can still crawl it. If you’d rather block an AI company completely, add its search crawler to the list too, like OAI-SearchBot or Claude-SearchBot.

The check for $is_public makes sure the rules are only added when the site is public. When “Discourage search engines from indexing this site” is checked in “Settings” → “Reading”, WordPress asks search engines to stay away with a noindex tag on every page instead.

One thing to watch out for: if there’s a real robots.txt file in the root of your site, WordPress serves that instead of generating one, and the filter has no effect. Check your root folder to make sure. If there is a file there, add the same rules to the file itself.

How to block AI crawlers on WordPress.com

If your site is hosted on WordPress.com, like mine, you don’t need any code. Every site, on every plan, has a setting that blocks known AI crawlers:

1. Go to “Settings” → “Reading”.
2. Under “Site visibility”, check “Prevent third-party sharing” below the “Public” option.
3. Click “Save Changes”.

Relying on the WordPress.com list of known AI crawlers means that you don’t have to stay abreast of new crawlers coming to market, or of existing ones being renamed. On the flip side, you are dependent on WordPress.com updating the list to block the crawler you want to keep out. Besides adding AI crawlers to robots.txt, the setting also keeps your content out of what WordPress.com shares with its third-party content and research partners. If you have more than one site, you need to turn it on for each of them.

On the Premium plan and up, WordPress.com also has a new “GEO” tab under “Jetpack” → “SEO”, where you can allow or block crawlers one by one, like in the code above. If “Prevent third-party sharing” is checked, it takes priority over the GEO tab for the crawlers it covers, so uncheck it if you want to make your own choices there. This feature is days old as I’m writing this, so it might have changed from the screenshot.

On the Business plan and up, you can also use the must-use plugin from the previous section, if you want full control over what is and isn’t excluded.

What robots.txt can’t do

robots.txt is a request, not a lock. The big AI companies say their crawlers follow it, but nothing stops a crawler from ignoring it, and some smaller ones do. I wouldn’t take the word of the big AI companies for it either.

If you want to stop crawlers from reaching your site at all, you need to block them before they get to WordPress. If your site is behind Cloudflare, its AI Crawl Control lets you choose separately whether to allow, disallow or block AI crawlers for training, for search and for agents. “Disallow” adds the rules to robots.txt for you, like the code above, while “Block” stops the crawlers at the door, whether they follow robots.txt or not. Many hosts have similar bot blocking at the server level, so check with yours.

Checking that it works

Open yoursite.com/robots.txt in your browser. You should see the AI crawlers listed with “Disallow: /” below them. If you don’t, a cache might still be serving the old version. Clear your site’s cache, and Cloudflare’s if you use it, and try again.

Crawlers don’t fetch robots.txt on every visit, so it can take a day or so before they notice the change.