: How to block a user-agent that has a spacing in its name? I got a hit from a crawler with a user-agent called DV CRAWLER which is an abvious a spam-bot. I tried to block it in both .htacess

@Ogunnowo487

The space is a delimiter (ie. a special character) in .htaccess so must be backslash escaped if you want to match a literal space in the regex. Eg. DV CRAWLER. (Otherwise you are likely to get a less than helpful 500 Internal Server error.)

Or, you can use the shorthand character class s which matches any white space character (space, tab or new line / line break) - so not technically just a space.

Vote Up Vote Down

@Heady270

When in doubt, add parenthesis and escaping to regular expressions. Try this first:

(Baiduspider|(DV CRAWLER))


I think that your problem is that it evaluating as "Baiduspider or DV followed by CRAWLER" when you don't have the paranthesis. If that doesn't work, then try escaping the space:

(Baiduspider|(DVsCRAWLER))


Where s is any white space character.

Vote Up Vote Down

@Sherry384

Your regex code in general is wrong.

Try instead something like this:

RewriteCond %{HTTP_USER_AGENT} (.*Baiduspider.*|.*DV.*CRAWLER.*) [NC]


You are matching against a string in each iteration between the parenthesis () separated by the pipe character | whereas .* is a wild card that matches anything. Optionally you can use s or s+ for spaces but .* works too and may be better. Not knowing what the DV CRAWLER string looks like, I made a guess (SWAG). You may need to adjust this.

For example: A string of a line of red cars driving down the street could be matched simply using .*red.*cars.*. There are slicker regular expressions for this, but using this simple method can be safely repeated over and over.

Vote Up Vote Down