: Page load speeds effect on crawl rate We've noticed a big drop in the total pages crawled per day on our site, we have no control over the crawl rate in Google Webmaster Tools so it's possible
We've noticed a big drop in the total pages crawled per day on our site, we have no control over the crawl rate in Google Webmaster Tools so it's possible this has been changed by Google. However it's a fairly large site and I wouldn't of thought that the crawl rate would've been decreased.
What we have noticed though is a sizeable increase in page load times, in my mind this would be the cause. Can anyone else confirm if the crawl rate is directly correlated to page load time? Seems logical, longer page load time, less pages crawled.
Any decent documentation on this would be appreciated, I don't normally have any input on SEO so this is new to me.
More posts by @Si4351233
1 Comments
Sorted by latest first Latest Oldest Best
I can confirm that Google crawls more when your site is faster and less when you site is slower.
In Google Webmaster Tools, the "Crawl rate" graphs will illustrate this perfectly. "Time spent downloading a page" will almost always be inversely proportional to the "Pages crawled per day" and "Kilobytes downloaded per day" graphs. Your Googlebot crawl allotment seems to be in terms of the total amount of time that Googlebot is willing to spend downloading from your site.
Here is a first hand report from a webmaster that improved his site performance which led to a lot more Googlebot crawling. The article includes this screenshot of the crawl rate graphs from Google Webmaster Tools that nicely illustrate the inverse relationship:
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.