The art and science of making your website rank high in the results of various search engines is commonly referred to as SEO (Search Engine Optimization). And when it comes to SEO, there are perhaps too many different aspects to cover in one article. That’s why we’re only going to focus on one right now: the WordPress robots.txt file. In this article, we will take a deep dive into what the robots.txt file is and how to use it. Among other things, we’ll discuss the different ways to create the file and cover best practices with its guidelines.
What is “robots.txt” file
Robots.txt is a text file located in your root WordPress directory. you can open it. yourwebsite.com/robots.txt URL in your browser. It lets search engine bots know which pages on your website should be crawled and which should not. Strictly speaking, a website does not need to have a robots.txt file. If you’re in the process of building your website, it’s best to focus on creating quality content first. Search engine bots crawl your website whether you have the robots.txt file or not.
However, when your WordPress website is finished, there are several benefits to having a robots.txt file with the appropriate instructions. Optimized robots.txt directives not only prevent crawlers from crawling unnecessary content, but also ensure that your crawl quota (the maximum time a crawler can crawl your website in a given period of time) isn’t wasted.
Also, well-written WordPress robots.txt directives can mitigate the negative effects of bad bots by not allowing them access. This, in turn, can increase your overall website loading speed. But keep in mind that robots.txt directives shouldn’t be your only security. Bad bots often ignore these guidelines, so using a good security plugin is highly recommended, especially if your website is experiencing issues caused by bad bots.
Finally, it is a common misconception that the robots.txt file may prevent some pages on your website from being indexed. The robots.txt file may contain instructions that prevent crawling, not indexing.. And even if a page is not crawled, it can be indexed via external links to it. If you want to avoid indexing a particular page, noindex Meta tag instead of instructions in robots.txt file.
How to use the “robots.txt” file?
After explaining what a WordPress robots.txt file is and what it does, we can consider how it is used. In this section, we’ll cover how to create and edit a robots.txt file, some good practices for its content, and how to test it for errors.
How to create “robots.txt” file
By default, WordPress creates a virtual robots.txt file for any website. Such a file might look like this, for example:
However, if you want to edit it, you have to create an actual robots.txt file. In this section, we’ll explain three ways you can do this. Two of them involve the use of WordPress plugins, while the third relies on the use of FTP.
With over 5 million active installations, Yoast SEO It is one of the most popular SEO plugins available. It has many site optimization tools, including a feature that allows users to create and edit robots.txt files.
After installing the plugin, click on newly created SEO sectionnext click Vehicles subsection. On the page that opens, click file editor connection Near the hill.
on the next page, robots.txt Section. From there, if you haven’t already created it, Press Create robots.txt file button.
This will create the file and you will be able to see its contents in a text field. You can edit the contents of your new robots.txt file using the same text field. When you’re done editing the file, Save changes to robots.txt file button below.
All in one SEO It is a hugely popular SEO plugin that comes with several free features, including one that allows users to create and edit WordPress robots.txt files.
After installing the plugin, click on new one All in One SEO section in your clipboard menu, then a click Feature Manager choice. In Feature Manager page, find your place robots.txt feature and then Press enable button next to him.
This will create the robots.txt file. After that, You will also see a success message stating that the options have been updated.. And a new subsection called robots.txt will appear.
by clicking on robots.txt option, you will get a view of the new partition. There you will be able to add new rules/guidelines to the robots.txt file and see how it currently looks.
Besides using a WordPress plugin, you can manually generate the robots.txt file. Firstly, create an empty .txt type file on your computer and save as robots.txt.
Then you need to upload it to your server using FTP. If you are not familiar with FTP, you should learn more about using FTP before continuing.
when ready, Connect to your server using your FTP credentials. Then, on the right, go to your root WordPress directory, is usually called public_html. on the left side of your FTP client (we Filezilla), Find the robots.txt file previously created and saved on your computer. Right click on it and choose To upload choice.
In a few minutes the file will be uploaded and you will be able to see it among the files in your root WordPress directory.
If you want to edit the uploaded robots.txt file later, locate it in the root WordPress directory, right-click on it and Edit View choice.
Adding rules to the “robots.txt” file
Now that you know how to create and edit the robots.txt file, we can talk more about the instructions this file might contain. The two most common directives found in the robots.txt file are: user agent and not allow.
The User-agent directive specifies which bot will apply the directives listed below the User-agent directive. You can specify a single bot (eg User-agent: Bingbot) or put an asterisk so that the guidelines apply to all bots (User agent: *).
The Disallow directive tells the bot not to access a certain part of your website. And there is also Allow directive which does the opposite. You don’t need to use it as often as Disallow, as bots are given access to your website by default. The Allow directive is often used with the Disallow directive. More precisely, it serves to allow access to a file or subfolder belonging to a disallowed folder.
In addition, there are two more directives: Scan delay and Sitemap. The Crawl-delay directive is used to prevent the server from overloading due to excessive crawl requests. However, this directive should be used with caution because it is not supported by some browsers (e.g. Googlebot) and interpreted differently by browsers that support it (e.g. BingBot). The Sitemap directive is used to direct search engines to your XML sitemap file. Using this directive is highly recommended as it can help you submit the XML sitemap you created to Google Search Console or Bing Webmaster Tools. Note, however, that you must use an absolute URL to reference your sitemap (e.g. Sitemap: https://www.example.com/sitemap_index.xml) when using this directive.
In the section below, we’ll show you two sample pieces to demonstrate the use of the robots.txt directives we mentioned above. But these are just examples; depending on your website, you may need a different subset of directives. Having said that, let’s take a look at the trailers.
User agent: *
Don’t allow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
User Agent: *
Allow: /wp-content/uploads/
Don’t allow: /wp-content/plugins/
Don’t allow: /wp-admin/
Don’t allow: /readme.html
Allow: /reference/
Sitemap: https://www.example.com/sitemap_index.xml
Testing the “robots.txt” file
You should test your WordPress robots.txt file after adding the appropriate guidelines for your website needs. By doing this, you not only verify that the file has no syntax errors, but also ensure that the appropriate areas of your website are properly allowed or disallowed.
To test your website’s robots.txt file, go somewhere SEO testing site. back then add any URL managed by your website (for example, the URL of your homepage), choose a user agent (e.g. Googlebot) and Press Scale button.
If the URL is crawlable, you’ll see a green result that says: Allowed. Otherwise, it will say Not allowed. You can repeat the same process for as many different URLs on your website as you want to confirm the proper crawling guidelines on your website.
final thoughts
The robots.txt file is a text file located in the root directory of every WordPress site. It contains instructions that tell crawlers which parts of your website they should or should not crawl. While this file is virtual by default, knowing how to create it yourself can be very beneficial for your SEO efforts.
That’s why we’ve covered various ways to create a physical version of the file and shared instructions on how to edit it. We also touched on the main guidelines a WordPress robots.txt file should contain and how to test if you’ve set them up correctly. Now that you have mastered all of this, you can consider how to rank for other SEO aspects of your site.
