: What would prompt multiple web crawlers to scan a site? I ask because a few of my servers are currently being scanned by 5+ bots: Bing, Google, Baidu, Sougu, and Yandex, along with Google's

@Gretchen104

Usually all it takes is a single link from one site to another. Once your site is discoverable via links, it will eventually be crawled as the spiders move throughout the web.

As for why you are just now seeing it, no idea. There could be a number of factors in play with this, including a simple frequency increase as the search engines decide that you have stuff that should be indexed more often.

If you are checking the IP addresses of the bots to be sure they are real and not malware masquerading as a bot you should be fine and there isn't anything to worry about.

Vote Up Vote Down