: How to determine if Google's Crawl Rate for your site is eating up too much Bandwidth? I was reading about changing Google's crawl rate and how there is a balance in which if the crawl rate

@Berumen354

Generally, if you have a popular website, you need to factor in to the server's performance capabilities, a) your customers/viewers that will be on simultaneously and b) being hit simultaneously by Yahoo, Bing, and Google updates, and possibly Yandex and Baidu, depending on your viewing target area.

Slowing down Bing and Yahoo via the robots.txt entry wasn't a good idea as it started affecting the number of indexed pages. They tend to hit your website harder for traffic that Google does, and Baidu and Yandex are worse at moderating their traffic when they pay your website a visit.

So I eliminated the entry in the robots.txt file (Google doesn't use it anyway) and use the Bing and Google webmaster tools settings to do some traffic shaping. What I finally settled on was to allow more than usual traffic on off-peak hours and ask for slightly less traffic during normal customer high traffic hours to keep the site responsive for them.

My information on traffic comes from my hosting provider's implementation of cPanel. They provide a traffic graph for daily, weekly and yearly bandwidth which I used to tune the settings. You can get a program like Weblog Expert to extract traffic statistics out of your access logs as well.

Vote Up Vote Down