: Can search engine robots read file with permission 640? I am on a shared web hosting linux server. I want search engine robots/spiders to be able to read the robots.txt but not any one typing
I am on a shared web hosting linux server. I want search engine robots/spiders to be able to read the robots.txt but not any one typing mysite.com/robots.txt.
As per the following google group post, the user specifies that by setting file permission to 640, it's possible to deny access to robots.txt file by the world but still enable search engine robots to read them.
Is that true? If not how it's possible to deny general public access to robots.txt but still allow Search engine robots to read them.
More posts by @Gretchen104
2 Comments
Sorted by latest first Latest Oldest Best
No, it's not true. Whether a bot or a person is accessing your site they all go through the webserver software (Apache for example) which accesses the robots.txt file in the same way for every request. It will either always be able to read it, or never. If it's never then something's probably configured wrong.
As long as your robots.txt is accessible by search engines on your server, any user can directly access it by typing its address.
You could block its access via .htaccess using specific user-agent filters, though, namely authorizing only search engines user agents to access it.
As long as no one is fooling around with a fake user agent to blind guess your robots.txt file, you should be safe.
The real question is: why don't you want your users to see it?
If any informaton is too important to be visible on the web, it should not be published online (unless behind a password-protected wall).
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.