: How To Slow Down A Generic Bot? There is a generic bot (only identified as 'bot*') consuming most of my bandwidth and processing power. Blocking its IP stops it but since it comes from a

@Shelley277

Posted in: #WebCrawlers

There is a generic bot (only identified as 'bot*') consuming most of my bandwidth and processing power. Blocking its IP stops it but since it comes from a well-known search-engine, I'd rather slow it down instead (it may be doing some useful experimental work). It's not the official crawler which actually visits my site a reasonable rate.

There is a similar question for GoogleBot which I have no problems with. The culprit takes 16X more bandwidth! What I need though is a way to slow down a bot which is only distinguishable by IP. Is this possible?

EDIT:

This is a Virtual Shared Host running Apache under Linux 64-bits with PHP 5.3.8. Although I have no direct control to the webserver, htaccess is enabled and I can have them add and configure modules on request.

Vote Up Vote Down

Login to post a comment!