How to edit robots.txt in WordPress

Websites use the robots.txt file to tell web crawlers like the ones from Google and other search engines what pages they are allowed to access. By default, the only change you can make to the robots.txt file in WordPress is through the discourage search engines feature, which either blocks or allows all traffic. You can’t edit the contents of robots.txt directly out of the box in WordPress, and that’s probably a good thing – it’s pretty easy to make a mistake and block crawlers from listing your website in search results.

If you do want to get your hands dirty with the robots.txt file in WordPress, you have three options: install a plugin, add a robots.txt file of your own with a FTP client, or filter the output of the robots.txt file with a PHP function in your theme.

Edit robots.txt with a plugin

One of the more popular and trustworthy plugins for editing robots.txt is Yoast SEO. Yoast SEO is first and foremost a SEO management plugin, but it also includes other nifty features like the robots.txt editor.

1. Log in to the administration panel on your site.
2. Click on “Plugins” in the menu on the left, and then on “Add Plugin”.
3. Enter “Yoast SEO” in the search field, and click on the “Install Now” button when it shows up in the search results.
4. Wait for the pending “Installing” button to change to “Activate”, and click it.
5. Yoast SEO will redirect you to its installation wizard. You can go through it if you want, or continue directly to the robots.txt editor by clicking on “Yoast SEO” → “Tools” in the menu on the left.
6. Locate the link to the “File editor” on the page, and click it.
7. If you don’t have a robots.txt file already, click on the “Create robots.txt file” button. Yoast SEO will create a robots.txt file that matches the virtual robots.txt file that WordPress generates based on the plugins you have installed.
8. Now, you can edit the contents of robots.txt in the text area. Click the “Save changes to robots.txt” button when you’re done.

Edit robots.txt with a FTP client

1. Open your FTP client of choice and connect to your website. I like Transmit.
2. Locate and navigate to the root of your site. If you find the wp-config.php file, you’re in the right place.
3. There likely won’t be a robots.txt file in the root of your website already, since WordPress uses a virtual robots.txt that it generates on the fly by default. If there is one, you can open it and make your changes in the file editor.
4. If there isn’t one, create a new file called robots.txt with your changes and save it to the folder. The file you add will be used in place of the WordPress defaults, so make sure that you do not make any unintended changes. You can see what WordPress generates by default by looking at the do_robots() function in WordPress Core. If all of this sounds like gobbledygook to you, you likely shouldn’t mess with it at all.

Edit robots.txt with a PHP function in your theme

Like most things in WordPress, you can filter the content of the virtual robots.txt file with a function in your theme or plugin. The benefit of this over the previous two methods is that your changes can be made in addition to the changes that WordPress and your plugins make to the file. You can also overwrite the changes made by Core and your plugins altogether, if you want to. If you track your theme in a GitHub repository, your edits to robots.txt will also show up in the repository version history, which can be useful.

1. Open the functions.php file in your WordPress theme, or a suitable PHP file in your plugin. Keep in mind that any changes you make directly to a theme from WordPress.org will be overwritten the next time you update it.
2. The hook used for modifying the contents of the virtual robots.txt file is called robots_txt. Add the following code to the PHP file you opened in the previous step:

/**
 * Modify robots.txt to disallow certain file extensions from being accessed by crawlers.
 * Replace the "slug" in the function name with an appropriate slug for your theme or plugin.
 * For the former, the slug is usually the same as the theme folder name (with underscores instead of hyphens in place of spaces).
 */
function slug_modify_robots_txt( $output, $public ) {
	/**
	 * Only make changes if "Discourage search engines" is disabled and the site is marked as public.
	 */
	if ( $public === true ) {
		/**
		 * Disallow some file types from being accessed by search engines.
		 */
		$disallowed_file_type_extensions = array( 'jpeg', 'jpg', 'gif', 'png', 'mp4', 'webm', 'woff', 'woff2', 'ttf', 'eot' );
		foreach( $disallowed_file_type_extensions as $extension ) {
			$output .= "Disallow: /*.{$ext}$\n";
		}
	}
	/**
	 * Return the output.
	 */
	return $output;
}
add_filter( 'robots_txt', 'slug_modify_robots_txt', 99, 2 ); // Priority 99, two arguments.

3. In this example snippet, we update the virtual robots.txt file to tell crawlers not to access certain file types. You can change this example to whatever you want, but as mentioned above, you should always be careful when dealing with robots.txt. The high priority value set for the add_filter() parameter means that the function will run after WordPress and SEO plugins have made their changes to robots.txt, and the use of $output .= means that we’re adding to those changes rather than replacing them completely.
4. Save and publish your changes to the site.