: Googlebot sends several hundred requests in a minute I have a website which has several million page views per month. Starting several days ago, a bot named Googlebot sends requests to the site

@Welton855

In your robots.txt file add a crawl delay directive.

User-agent: *
Crawl-delay: 1

Any IPs that don't obey this should be blocked.

Anything in the 66.249 range is Google in my understanding.

Vote Up Vote Down

@Harper822

It's possible those accesses are page redirects or supplementary document files such as images. Perhaps each of your pages has a large number of assets attached to it that must load in order for the page to be complete, such as external javascript files, CSS files, and image files.

Like closetnoc said, you can try google search console to limit crawl rate, but if that isn't good enough, then consider issuing a 410 status code to the requests that refer to old pages that no longer exist so that google request those pages less often, and redirect some old urls to new urls if the redirect makes sense and helps your guests.

If google is still ridiculous, then access your server firewall and limit the requests google can make by filtering its IP address block, but I don't recommend stopping google from making 10 requests a second.

Vote Up Vote Down

@Alves908

The example IP addresses are Google alright! However, it should be no more than 1 request per 2 seconds though that may have changed in the past few years. For the record, Google has always behaved rather well and has always stayed within the 1 request per 2 seconds rule. Still, it is not uncommon for small flurries of requests throughout the day and maybe larger flurries of requests during non-busy times.

The average you quoted from Google is for all sites across the Internet. I have had as many as 49,000+ pages crawled per day and I am sure more when I was not paying attention. Google pings my site by the 10's of thousands regularly and only pauses a few days at a time. The longest Google has stayed away was 5-6 days. It all depends upon the site size, freshness, popularity, and site speed.

You can control the crawl rate but only temporarily. Sheesh! You can set the crawl rate using Googles Search Console (webmaster tools), however, Google only honors that rate for 90 days. Google will begin to crawl your site at a safe rate according to your site speed as normal.

Most people want Google to go faster.

Most people want to increase Google's crawl rate. Personally, I never cared. However, if you want to slow Google down, then there are only two solutions. The first is Search Console. You can find out how at the bottom of this page: support.google.com/webmasters/answer/48620?hl=en You will also see mention of the other option which I do not advise as it may be permanent. That option is filing a request here: www.google.com/webmasters/tools/googlebot-report Both options requires an account.

Do not count on the crawl delay directive to help you. It is not clear that Google pays attention to it. In fact, if I were a betting man, I would bet against it. Google should pay attention to this directive. It is part of a standard they support.

Vote Up Vote Down