: Why does bingbot access my sites the most even if I set crawl rate to the minmum? Recently my sites got overloaded and it turned out bingbot was the culprit. I set the crawl rate for the

@Gloria169

Are you sure it is really a Bing bot? I have seen bots that ignore disallows that turned out to be from somewhere other than expected when doing a reverse IP lookup using dig.

dig -x 157.55.39.142

; <<>> DiG 9.9.5-3-Ubuntu <<>> -x 157.55.39.142
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30244
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;142.39.55.157.in-addr.arpa. IN PTR

;; ANSWER SECTION:
142.39.55.157.in-addr.arpa. 71 IN PTR msnbot-157-55-39-142.search.msn.com.

;; Query time: 57 msec
;; SERVER: 8.8.4.4#53(8.8.4.4)
;; WHEN: Thu Jun 12 18:18:06 EDT 2014
;; MSG SIZE rcvd: 104


The reverse lookup shows the IP is indeed related to Bing's search bots.

They do state that the crawl control directive in a robots.txt file will take precedence over a configured setting on their end.
www.bing.com/webmaster/help/crawl-control-55a30302
If that doesn't seem to be the case you may want to try using their controls to adjust their crawl for your site.

Vote Up Vote Down