: Wget not respecting my robots.txt. Is there an interceptor? I have a website where I post csv files as a free service. Recently I have noticed that wget and libwww have been scraping pretty

@Kristi941

If you decide you want to block wget and libwww you can either redirect them to a page telling them why you're blocking them with this code:

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^libwww
RewriteRule ^(.*)$ www.example.com/blocked.html

Or you can flat out reject their request with this code:

SetEnvIfNoCase user-agent "^wget " bad_bot=1
SetEnvIfNoCase user-agent "^libwww" bad_bot=1
<FilesMatch "(.*)">
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</FilesMatch>


Just place either snippet in a .htaccess file in your root directory or the directory where the files they are downloading from are.

I have used the second snippet to block bots from a site I had that was getting scraped a lot. I haven't used the first snippet but it looks like it should work well if you choose to go that route.

Vote Up Vote Down