: Why do some Nginx configurations block download agents using Wget? I noticed that some Nginx configurations block download agents using Wget like this: if ($http_user_agent ~* LWP::Simple|wget) {

@Turnbaugh106

User agent wget

Blocking the user agent wget will block operating systems that use wget to download files within the terminal i.e Linux, MacOS, Arm etc. This will block every file from being downloaded using wget example.pdf.

But why block wget?

There's no right or wrong to this question but generally webmasters want site visitors actually visiting their pages to download their stuff, this is among one of many hot linking prevention methods.

What will get blocked?

Only the user agent wget associated with the terminal application as described previously all file types will be prevented including .html files. Browsers will not be affected unless they use the wget user agent which by default they don't and only knowledgeable users will be able to achieve this but users have no benefit to user that user agent so realistically a block will never occur in anyone's browser.

When wget is useful

It's not very useful for users to use wget to download a PDF file or video clip unless they want to mirror it or because the file size is very large and they want to store it somewhere remotely. However it can be useful for things like deployment of content management systems such as Joomla and Wordpress which have several hundred of files each and using:


wget www.example.com/cms.tar
tar xvf cms.tar


By doing the above actions you can deploy a content management system within seconds rather than minutes vs using SFTP or FTP method due to the limitation of the protocols. So wget shouldn't be used for things like operating systems, content management systems and so forth but generally its useless for most websites so blocking it shouldn't become a major issue.

Vote Up Vote Down