Mobile app version of vmapp.org
Login or Join
Nimeshi995

: Htaccess empty referer deny "google bot" I put this rule into htaccess file to deny empty referers which is returning 403 SetEnvIfNoCase Referer "^$" bad_user Deny from env=bad_user I

@Nimeshi995

Posted in: #403Forbidden #Googlebot #Htaccess #WebCrawlers

I put this rule into htaccess file to deny empty referers which is returning 403

SetEnvIfNoCase Referer "^$" bad_user
Deny from env=bad_user


I can see the log, it's denying also Googlebot which is acting as empty referrer also. Is there a way to modify the rule to allow access for Googlebot and deny all the rest empty referrers?
I have blocked one referrer example.com using

RewriteCond %{HTTP_REFERER} example.com [NC]
RewriteRule .* - [F]


It's working well, returning 403, but what if he is referring from example.com/another_page?

So I did this:

RewriteCond %{HTTP_REFERER} example.com [NC,OR]
RewriteCond %{HTTP_REFERER} example/another_page.com/
RewriteRule .* - [F]


Is it correct?
How can I block this user agent: Mozilla/5.0/Firefox/42.0 - nbertaupete95(at)gmail.com?
What should the rule look like?
Will this one work?

RewriteCond %{HTTP_USER_AGENT} ^nbertaupete95(at)gmail.com [NC]
RewriteRule .* - [F,L]

10.01% popularity Vote Up Vote Down


Login to follow query

More posts by @Nimeshi995

1 Comments

Sorted by latest first Latest Oldest Best

 

@Alves908

As you have found (and warned against in comments), you should not try to block requests that contain an empty Referer header. Google (and most bots) and many legitimate users will not send the HTTP Referer header (at least at some point), so this will just cause you problems.

To answer the two remaining queries in your question...



I have blocked one referrer example.com using

RewriteCond %{HTTP_REFERER} example.com [NC]
RewriteRule .* - [F]


It's working well, returning 403, but what if he is referring from example.com/another_page?



You don't need to change anything to your existing directive(s). The first RewriteCond directive already blocks any referer that simply contains example.com. Note that example.com is a regex (regular expression) (which is why the dot is backslash escaped). Without any anchors, the pattern is naturally matched anywhere inside the HTTP_REFERER.

(However, the pattern in your additional directive example/another_page.com/ is a bit mashed anyway, so would never match as intended. That wouldn't have mattered, since it will have matched the first condition anyway.)



How can I block this user agent: Mozilla/5.0/Firefox/42.0 - nbertaupete95(at)gmail.com? What should the rule look like? Will this one work?

RewriteCond %{HTTP_USER_AGENT} ^nbertaupete95(at)gmail.com [NC]
RewriteRule .* - [F,L]




No, this would not work, the regex is not correct. You have introduced a start of string anchor (^) on the regex, so it will only match user-agents that start "nbertaupete95...". In the stated user-agent string, this substring is contained within it.

Also, parentheses (( and )) are special meta characters in regex - they are used for alternation and to capture subpatterns. To match a literal parenthesis, these must be backslash-escaped, eg. (.

So, your example should read something like the following instead:

RewriteCond %{HTTP_USER_AGENT} nbertaupete95(at)gmail.com
RewriteRule .* - [F]


The L flag is not required when using F (it is implied). The NC flag on the _CondPattern_ is also unnecessary, unless this user-agent really does have case variation? Also, dots must be escaped to match a literal dot, as you did in your earlier directives.

In the stated user-agent, the "email address" occurs at the end of the user-agent. If this is the case, then you can add an end-of-string anchor ($) to the regex, for example: nbertaupete95(at)gmail.com$.

10% popularity Vote Up Vote Down


Back to top | Use Dark Theme