: Multipe Bots Simultaneously Hitting Site Have had a very strange set of httpd access logs today from before a pagespeed mod and apache crash. It seemed that the following bots appeared to
Have had a very strange set of httpd access logs today from before a pagespeed mod and apache crash. It seemed that the following bots appeared to be indexing the site, like crazy, all at once.
googlebot
googlebot-image
bingbot
yahoo! slurp
yandexBot
Baiduspider
Sogu Web spider
AhrefsBot
SeznamBot
msnbot
Plus a couple more I can't locate at once.
This looks highly suspicious to me. In amongst the bot requests there are some odd URL requests that are not in the site, so are not being spidered (things like word press admin URLs and it not even a wp site).
The bot IPs look genuine. Anyone know how someone might be able to trigger a bot-storm? Never seen that before - seems very odd indeed, can't believe this is mere coincidence.
More posts by @Deb1703797
1 Comments
Sorted by latest first Latest Oldest Best
There are occasions where a large number of search engines will try to index your site at once. As has been mentioned in the comments this can be from one or more of the large number of ping tools on the web which tell all search engines out there that there is new content to be indexed.
You have done the right thing in verifying the IP's as resolving to Bing and Google as that is the most important thing to ensure that the crawling is coming from a legitimate location.
It isn't necessarily something to worry about, sometimes by pure chance a large number of search engines may try and crawl your site at the same time, especially if someone adds a new link to your site for a page on your site that the search engines don't know about yet.
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.