: Understanding the maximum hit-rate supported by a web-server I would like to crawl a publicly available site (and one that's legal to crawl) for a personal project. From a brief trial of the

@Nimeshi995

There is no standard. You will have to consider how robust the site is. Multiple servers and lots of bandwidth would be less taxed than a single server with less bandwidth. Also consider that some are paying for the bandwidth they use rather than having a set amount rated on a per second basis (like we are used to with DSL and cable). This would mean that your activities can really add a significant cost for the site.

Now for the bad news. The average hit rate of a human is .8 seconds per request. Google and Bing state that they try not to make more than one request every 2 seconds. As well, some systems block users with high access rates. If your spider efforts are spotted and the webmaster feels that you are inappropriate, you can be blocked extremely quickly and effectively.

The best policy is to check the robots.txt file to see if there is a crawl rate and respect the wishes of the site owner. As well, it is often best to contact the site owner and ask for permission. Just because the site is publicly available does not mean you can abuse it. There is no assumption under the law (U.S.) that gives you automatic rights to a network or the resources offered. This means that a civil case can be filed for abuses that can be clearly documented.

I can tell you this. Most sophisticated spider blocks will not even tolerate your 4 access per second rate. You would be blocked and a manual process would have to occur to remove the block.

I will not even mention anything about copyright infringements and your lack of rights to original content. Get permission. You may get what you want.

Vote Up Vote Down