: Robots.txt blocking specific files also block unnecessary URLs I am using Magento for one of my site. In Magento there is a file mage (no extension file name is mage only) to block this file

@BetL925

You can add a dollar sign to the end of the string which means it will only match exactly that entry:

# Files
User-agent: *
Disallow: /mage$


This will only block the mage file if it come straight after the root domain:


example.com/mage

If there are any other preceding directories, you must add these o the entry. So to block the file located below:
example.com/somedirectory/mage

You would need to use:

# Files
User-agent: *
Disallow: /somedirectory/mage$

Vote Up Vote Down