: How do I block a user-agent from Apache How do I realize a UA string block by regular expression in the config files of my Apache webserver? For example: if I would like to block out all

@Vandalay111

I enabled rewrite engine in Apache:

a2enmod rewrite


and added this block to my /etc/apache2/httpd.conf

<Directory /var/www/>
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} googlebot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} sosospider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} BaiduSpider [NC]
# Allow access to robots.txt and forbidden message
# at least 403 or else it will loop
RewriteCond %{REQUEST_URI} !^/robots.txt$
RewriteCond %{REQUEST_URI} !^/403.shtml$
RewriteRule ^.* - [F,L]
</IfModule>
</Directory>


and restarted Apache:

apache2ctl graceful


now these calls from those spiders all cause 403 Errors:

grep -E 'spider|bot' /var/log/apache2/*.log

Vote Up Vote Down