: Can I block requests that have no user agent because they cause 404 errors? Is it OK to block requests that contain no user agent? I am getting random 404 errors on URLs I have never used.

@Turnbaugh106

Blocking No User Agent

Blocking based on no user agent is a silly idea... a lot of users who like to remain anonymous through VPNS will often disable user agent and anything else that can be used to harvest data... And anonymity is growing. Also if the idea behind this is to save on resources it should be noted that most bots that are not legit search engine crawlers use user agents.

404's is normal

The correct code for a page that doesn't exist is 404... returning anything else would be incorrect and if your having issues with bots and concerned that a lot of resources are being consumed then using block on user agent would be incorrect through it would work but why when there's a simpler way...

Slowing down requests

You could block IP's, User Agents and so on but there's a much easier way... simply by using status 429 Too Many Requests (RFC 6585) this tackles all bots, all users and so on that request too much data too often within a certain period. It's common to see that some of the worlds most visited sites use this method and its effective. If you want to spend your time going through IPs from logs and then ban them then the status code you should return 403 Forbidden

Vote Up Vote Down