When you run a website, you want it to rank as high as possible in search engine results pages (aka SERPs). To achieve this goal, you need to use all the optimization techniques in your arsenal and do your best to come up with innovative ways to convince search engines that your content is relevant and valuable.
The process of getting your website to the top of the SERPs begins with the indexing of its own pages. Allowing bots to access your content means your pages are ready to be shared with the world.
Of course, you can always lock your pages with a password if you don’t want users to access it. However, this does not hide these pages from search engines.
But truth be told, there are many times when it’s better to noindex a page in WordPress. And to hide it from search engines, you need to tell them not to index it. This is done using the noindex directive, which allows bots to crawl a page but prevents them from ranking.
In this article, we’ll learn how to noindex a page in WordPress and explore:
Let’s start now.
While this may seem strange, sometimes it’s perfectly reasonable to noindex a page in WordPress. In fact, we’ll give 6 reasons why this might be true.
1. Privacy
you don’t want to share pages containing sensitive information, confidential details, or users’ personal and contact information with the people.
2. Duplicate Content
In case you publish your content in several formats (for example, Adobe PDF or printer-friendly versions), we recommend that you: show only one version to search engines To avoid duplicate content issues. Also, if you run an e-commerce site, you should always hide pages with similar or identical product descriptions (for example, only if product dimensions or colors change).
3. Keyword Cannibalization Issues
If you target a keyword on multiple pages or posts on your website, the page you think is most relevant for a particular query may not be the page that will rank high in search engines. To avoid this problem, enable indexing only for pages that are most important and relevant to specific query terms.
4. Little or No User Value
Thanks pages, admin login pages, and similar types of pages are not that important to users, so they don’t need to rank in search engine results.. Your website has no real value to your visitors and you don’t need them to get organic traffic.
5. Work in Progress
in your case pages and posts are under development, you should hide them from search bots until you perfect them to your liking.
6. Third Party Content
If your website features ads and similar content generated by third-party sourcesYou should hide it from search engines. This type of content usually has nothing to do with the primary purpose of your website.
Well, we know when we should hide a WordPress page or post from search engines. Now it’s time to learn how to do this.
Hiding your content from search engines using Yoast SEO is the easiest option to track, no matter your level of WordPress expertise.
To use this method, you need to install and configure the Yoast plugin. Next, access the post or page you want to hide from your WordPress dashboard and look for the Yoast SEO meta box (it should be just below the page editor). Now, click Further.
Several options will appear, but we are interested in the first two. First, it allows you to prevent search engines from showing your page in search results, and second, it prevents search engines from following links on your page.
adding following Tagging your page is usually not necessary. In case you can use paid links on your page or user-submitted links in the comments section. The first is sponsored content and Google’s Webmasters Guidelines. As for the latter, the problem is that you never know how reliable users’ links are, so it may be best to prevent bots from following them.
Other than that, just set it to noindex to keep your pages out of the search index.
When you’re done making changes to Yoast, don’t forget to save them.
This method is not recommended for WordPress beginners as it requires you to modify the robots.txt file.
Connect to your server via FTP to access the file or File Manager in your cPanel.
Another way to access the robots.txt file is to use the Yoast SEO plugin. go SEO > Tools and choose File Editor.
If you have a robots.txt file, it will be displayed at the top. If you haven’t, you can make one yourself. Google has a great article How to create robots.txt file. To hide a page from search engines via the robots.txt file, add the following lines of code to the file:
User-agent: * Disallow: /the-URL-of-your-page/
The first line of code states: specific bots and search engines you might want to target. Use an asterisk to include all search engines, just like we did. Add the URL of the page you want to hide in the second line of code, but be sure to exclude your website domain..
For example, the URL of your page “http://aweekendinthecity.com/2020/music/indie-rock” you would put:
User-agent: * Disallow: /2020/music/indie-rock/
When you’re done, you can Test your robots.txt file, just to make sure you set everything right. And if you just created the file, make sure to upload it to your website root directory.
Risks of Using the robots.txt File to Hide Content from Search Engines
Aside from not being quite suitable for beginners, this method comes with a few risks, so we’re not its biggest fans.
The robots.txt file is a configuration file that tells search engine crawlers whether they should access your pages and posts.. However, not all search engines follow requests for this file. It’s possible that your page will continue to appear in search results without any meta descriptions. Also, if another website is pointing to your site with descriptive text, your page will be indexed (even if it only contains audio and/or video content). Also, malicious bots and content scrapers often ignore the file.
The robots.txt file is also available to everyone.. This means anyone can see it and easily discover the pages you want to hide.
Another way to hide your page from search engines is to add a noindex meta tag to your page’s HTML file or add a noindex header to the HTTP response headers. For this to work, Google insists: don’t block your page with a robots.txt file otherwise search engine bots won’t even see the noindex directive.
How to Add ‘noindex’ Directive as Meta Tag
Add the noindex meta tag to the HTML to instruct all search engines not to index your WordPress page. <head> section of a specific page:
<meta name="robots" content="noindex" />
Prevent just Googlebot use this tag to crawl your page:
<meta name="googlebot" content="noindex" />
NB Even after changing the section and adding a noindex meta tag to it, some search engines may continue to crawl your page.
How to Add ‘noindex’ Directive in HTTP Response Header
To instruct search engines not to index your page, you can use the X-Robots-Tag and add the following HTTP response in the header of the header.php file:
header("X-Robots-Tag: noindex", true);
If you also want to stop bots from following links on your page, go with this HTTP response:
header("X-Robots-Tag: noindex, nofollow", true);
Any directive you can use in a robots meta tag can also be applied to an X-Robots-Tag. But the latter is much more flexible. Actually, X-Robots-Tag controls the indexing of both an entire page and specific elements on a page, while a robots tag allows you to manage indexing only at the page level. For example, if you want to stop search engine crawlers from indexing just one image or video file, you should use the X-Robots-Tag.
Which Method Should You Choose?
After all is said and done, We recommend using the Yoast SEO plugin to successfully noindex a page in WordPress. It is simple and effective. No advanced expertise is required to use it. Also, changes you make with Yoast do not affect your site’s root directory.
However, try other options if you want. In the end, all that matters is that you manage to prevent search engines from accessing and indexing your WordPress page or publishing it using whatever method you feel most comfortable with.
Support our work ❤️
If you enjoyed this article, consider leaving a tip to help us keep publishing great content.
