Google is incredibly good at doing many of the things we find necessary to run our daily lives. It helps us find information. It helps us to find the nearest store with what we want. It helps us get from point A to point B with as little fuss as possible.
One of all the things Google’s search engine does well is crawling a website. Google’s crawlers are extremely good at what they do, indexing your website so it can present in search results.
Sometimes, however, you can ask Google to re-crawl your website – go through a single page or the entire website once. If that’s what you want to do, we’ll teach you how. In this article you will learn:
Web crawling is the process by which a web spider, also called a spider bot, discovers and controls pages on your website. Web crawling is an essential part of a search engine’s job, as is any other important activity that makes your website appear in search results.
Typically, a search engine like Google uses crawlers to “read” the pages it can index – it collects and stores data from them. This allows them to present these pages as search results when someone searches on Google, and the engine’s algorithms believe your website may have a valid response to their query.
In practice, we see this happen every day when you Google and type in what you are currently interested in. Results pages are filled with possible answers based on how Google’s algorithm ranks them, but are powered by the search engine’s ability to crawl and index information on your website.
The crawling process begins when Google receives the address of your page. This can happen in several ways – for example, Google might find a link to it on another website. Your managed hosting provider can also signal to Google, and you can do this yourself by providing a sitemap, which we’ll come back to later.
The search engine sends its crawlers, which find and analyze the content and code of the page. It then indexes the page unless there’s something preventing it and brings it up when someone searches online. Her almost as simple as that.
Crawling, like indexing, is a process with some nuances. For example, you can opt out of the process and tell Google not to crawl your website. You can also point to specific pages that you don’t want Google to crawl. You can even prevent Google from scanning resources like images or media files. All of this is managed by creating a robots.txt file and placing it in the root of your site.
However, telling Google not to crawl a page does not prevent it from appearing in search results. If Google’s spider finds your page through a link on a third-party website, it can still index it and have a version of the page appear in search results. For example, if you want certain pages such as admin login pages not to be found in Google search results. or thank you pages – you should use the noindex meta tag on it.
The main reason people ask Google to recrawl websites is because they have made some changes to the pages and they want Google to reflect them as soon as possible. It’s common to request a website-wide recrawl after a major content audit. You may also want this even after a rebrand or if, for example, you have switched from a blog WordPress theme to a magazine WordPress theme.
As for single page requests, you may want to use them when posting time-sensitive content and be the first to use that content in search results.. It’s also common to request a single page recrawl after a major or extensive change to a page’s content.
Everything in order. Make sure you don’t block pages you want crawled with a robots.txt file. Search engines have robots.txt testing tools that can help you figure out if a page is protected from crawling. You can find these tools in the Search Console for Google. If there are no obstacles there, you can proceed to the next step.
There are two ways you can ask Google to recrawl your website. However, in both cases, you can wait a while for the crawler to do its job and for your pages to be indexed and ready to appear in search results. Sending multiple requests won’t help with this.
To have Google recrawl your website, or generally if you want it to recrawl many URLs at once, you need to submit a sitemap to Google. But first, you need to create a sitemap; this is where you have a plugin like below. Yoast SEO It works. Set route to SEO > Generaland then find the XML Sitemaps option.
Make sure it is set to “On”, click the question mark next to it, press the hyperlinked text “see sitemap” and you will have access to your website’s sitemap. Once you have it, you can go to Google Search Console and add the website if you haven’t already.
The Sitemaps Tab is where you want to go – it allows you to enter the URL of your sitemap. If you have done this successfully, you will see the sitemap appear in the list below. Once this is done, Google should have no problem crawling your website.
Alternatively, you can navigate to the robots.txt file and specify the path to your sitemap. It should look something like this:
Sitemap: http://example.com/my_sitemap.xml
You can use an FTP client or trust Yoast SEO again to access the robots.txt file on your website. Set route to SEO > Tools from your website backend and then use the File editor to access it.
In some cases, you want Google to pay special attention to only a few links. The correct way to handle this is to use a URL Inspection tool on the URLs you want to inspect. The tool allows you to request recrawls, but remember – if you have multiple URLs that need to be recrawled, you should submit a sitemap instead.
Requesting a recrawl of a single URL is as simple as inspecting it with the URL Inspection tool in Google Search Console and selecting the Request Indexing option. Unless the page has obvious issues preventing it from being indexed – something you’ll know right away – the page will be added to the indexing queue.
Let’s wrap it up!
Your website’s visibility in search engines depends on the engine crawlers that create an inventory of your website’s content. If you’ve added new sections to the website, changed some old ones, or just come out of the dark and want people to find your website using search engines, you have two options.
Submitting a sitemap is a surefire way to get your entire website indexed, and you should use it if you have more than a few URLs that need to be indexed. Otherwise, you should use the URL Inspection tool to pinpoint the URLs that need to be reindexed and let Google know to do so.
Support our work ❤️
If you enjoyed this article, consider leaving a tip to help us keep publishing great content.
