: What if Google Crawl Rate increased Suddenly? We have observed that our website crawl rate has suddenly increased, we have around 1200 pages on the website but google bot has crawled around

@Harper822

We have around 1200 pages on the website but google bot has crawled around 2600 pages. ... is it OK if crawl rate increases suddenly?


It depends.

For starters, let google and the rest of the search engines crawl your website, especially if its a website that generates profit for your company (for example, an online store). I'll call search engines SEs from this point on.

SEs crawl websites at a rate it believes its fair to the web page servers and to guests relying on SEs for information.

SEs need to crawl sites often in order to determine what the website is about and they would like to provide guests with relevant information as often as possible.

If the crawl rate is too slow, then SEs may show outdated information about a web page to the user when users browse through search results which may show an unpleasant experience to the user.

If the crawl rate is too fast, then the web page server will be too slow in processing requests which also gives users an unpleasant experience while trying to navigate the pages from the affected server. It's possible your situation may be in this category, but with the speed of computers now, the delay may become lower.

The reason why google crawled your site like crazy is because any (or even all) of the following happened:


You adjusted the crawl rate to maximum in google search console settings.
You made significant changes to a large number of pages.
Your cache settings are terrible. Consider setting up HTTP cache-control so images are cached for at least 1 week and the static HTML should be cached for at least an hour.
Your webpage has a large number of assets or objects that are required to be loaded from the same server to make the page complete. P.S. Google has an image search and may index your images as well.


Make relevant changes then use webpagetest.org to test your pages to ensure that a large number of resources aren't constantly being fetched. Make sure you configure webpagetest.org to test the same page multiple times. You're good when you see on subsequent requests that the same static resources aren't being fetched again (example: images).

Now look at it this way... if you revamped your site and google only crawled one page an hour, then there's something wrong with google's site, or there's something wrong with your server, or you could have violated google's guidelines to the point where it doesn't trust that your site could provide useful legit content to guests.

Vote Up Vote Down