: Does Yahoo Slurp crawl hammer a site? A lot of robots.txt examples seem to feature Crawl-Delay for Yahoo Slurp for up to 2 hours, is this because Yahoo Slurp tends to hammer a website and

@Si4351233

Yes, from personal experience, Slurp is aggressive. Baidu even more so.

If your website can withstand Google, Bing, Slurp and Baidu indexing your site within the same week while serving your customers or content viewers with minimal lag, you've chosen your platform specifications and software setups quite well.

Vote Up Vote Down

@Kristi941

"Hammer" is subjective, but there does seem to be a general belief that Slurp is let's say...maybe a little more aggressive than it really needs to be, and that it's been going on for years, ie. 2007, 2009, 2011. A basic search for something like slurp robots delay will turn up a giant pile of complaints.

In case you're tempted to actually block Slurp, as suggested at one of those links, you might want to also see this recent question here for a different take.

Vote Up Vote Down