: Handling bots that request URLs in paths In my server log, I found at least one IP address to be requesting a full URL in an awkward place. For example, the header the client sends to my

@Eichhorn148

The HTTP 1.1 spec is very clear that

GET /path/to/resource HTTP/1.1
Host: example.com


and

GET example.com/path/to/resource HTTP/1.1


are equivalent requests. This is because the request starts with Request-Line which is defined as Method-Token Request-URI Protocol-Version and the Request-URI can be absolute: "*" | absoluteURI | abs_path | authority.

You should not try to configure your web server to respond differently to the different formats of requests. You would be breaking the spec. While browsers today typically use the former request format, there is no guarantee that they will continue to do so in the future. You don't want your website to suddenly stop working with the latest version of some browser.



You should instead ensure that your server does not serve content for unknown hosts. A request for any third party site should return a 404 not found (or possibly even 400 bad request). Bots that request third party sites are typically testing for open proxy servers.

One way to configure your web server to do so is to configure the first (default) virtual host to return a 404 page. Every legitimate site would be in a later virtual host directive.

Vote Up Vote Down

@LarsenBagley505

These happen all the time. I see this at least a dozen times a day in my server logs. Best bet is to block the connection from coming in at the firewall or gateway and that way it doesn't hit your server, otherwise if it isn't a big deal for you and isn't causing you too many hassles and you aren't seeing other errors in relation to this connection then you can pretty safely ignore it.

Vote Up Vote Down