: GoogleBot crawling a page up to 3,000 times per day For a website, GoogleBot daily crawls one particular URL more than one thousand times (mainly between 1,500 and 3,000, sometimes much more).

@Steve110

Posted in: #Google #Googlebot #Seo #WebCrawlers

For a website, GoogleBot daily crawls one particular URL more than one thousand times (mainly between 1,500 and 3,000, sometimes much more).

The Googlebot IP is a real Google one (for example : 66.249.76.21).

This strange (?) behaviour has lasted (at least) for two years. We have recently turn from HTTP to HTTPS, but it is still the same...
The first time I met this behaviour, the web page contained a form: the GoogleBot used to send thousands GET and POST requests during a very short period, then stopped suddenly these requests when the form was canceled... but later, it tried again from another page with a form.

When I forbid this URL to the robot, after a short period, another URL is "chosen" by the GoogleBot and crawled again and again (and often in the same period during the day). The last "highly crawled" does not contain any form.

A few days ago, I made another test: the highly crawled URL sends a 302 to the home page.

Results: the GoogleBot follows the 302: but does not follow any files or links from the home page: it sends again and again the same GET request everyday more than 1000 times.

The results is that the main part of the crawl budget is wasted to this single url... Other Robots have not this behaviour.

Have you ever met such a singular observation? Could it be a kind of "security" test?

Any idea about what to do?

Vote Up Vote Down

Login to post a comment!