: Why is this mobile/image Googlebot crawl rate so fast? (10-30 req per sec) This morning in logs I noticed a user got rate limited by our throttler 3 times within 1 second (429 too many requests
This morning in logs I noticed a user got rate limited by our throttler 3 times within 1 second (429 too many requests error) at 3:30:11 AM EST: Mozilla/5.0 (Linux; Android 6.0; GCE x86 phone Build/MRA59G.MZC35) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2679.0 Safari/537.36 at IP 104.196.83.207. I've only seen Google on this IP once before, so I'm unsure what it's purpose is.
Upon looking it up it seems it's a Googlebot indeed. Noted below by Analog, its coming from Google Cloud. The last paths it was trying before being limited for 30 seconds were images. We have never had problems with Google speeding the site until last night. Our throttler is set to trigger a 30 sec 429 error if a user hits 10 requests in a 5 second window (session var based). 3 logs imply they hit this limit 3 times within 1 second (30 total requests per sec).
Right now, the Search Console setting says it's on the "let Google decide" -- clicking the radio to change rate puts it square in the middle at "1 request every 10 seconds". On its maximum setting it allows 2 requests a second (ie throttle zone).
I will adjust the crawl rate manually to 1 every 10 seconds, but in the meantime why is this bot crawling so fast, and why is it on android mobile crawling only images?
More posts by @Correia994
1 Comments
Sorted by latest first Latest Oldest Best
It is my understand that all google bot crawler IP addresses can be identified by the RDNS which is set to something along the lines of "crawl-(IP ADDRESS).googlebot.com".
The useragent you mentioned is also not used by the googlebot. Google lists the useragent strings of all their bots here.
The IP address you posted is part of the netblock used by Google Cloud customers as seen in the ASN comments
NetRange: 104.196.0.0 - 104.199.255.255
CIDR: 104.196.0.0/14
NetName: GOOGLE-CLOUD
NetHandle: NET-104-196-0-0-1
Parent: NET104 (NET-104-0-0-0-0)
NetType: Direct Allocation
OriginAS: AS15169
Organization: Google Inc. (GOOGL-2)
RegDate: 2014-08-27
Updated: 2015-09-21
Comment: ** The IP addresses under this netblock are in use by Google Cloud customers **
Comment:
Comment: Direct all copyright and legal complaints to
Comment: support.google.com/legal/go/report Comment:
Comment: Direct all spam and abuse complaints to
Comment: support.google.com/code/go/gce_abuse_report Comment:
Comment: For fastest response, use the relevant forms above.
Comment:
Comment: Complaints can also be sent to the GC Abuse desk
Comment: (google-cloud-compliance@google.com)
Comment: but may have longer turnaround times.
Comment:
Comment: Complaints sent to any other POC will be ignored.
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.