: Disallow robots.txt from being accessed in a browser but still accessible by spiders? We make use of the robots.txt file to prevent Google (and other search spiders) from crawling certain pages/directories
We make use of the robots.txt file to prevent Google (and other search spiders) from crawling certain pages/directories in our domain. Some of these directories/files are secret, meaning they aren't linked (except perhaps on other pages encompassed by the robots.txt file). Some of these directories/files aren't secret, we just don't want them indexed.
If somebody browses directly to mydomain.com/robots.txt, they can see the contents of the robots.txt file. From a security standpoint, this is not something we want publicly available to anybody. Any directories that contain secure information are set behind authentication, but we still don't want them to be discoverable unless the user specifically knows about them.
Is there a way to provide a robots.txt file but to have it's presence masked by John Doe accessing it from his browser? Perhaps by using PHP to generate the document based on certain criteria? Perhaps something I'm not thinking of? We'd prefer a way to centrally do it (meaning a <meta> tag solution is less than ideal).
More posts by @Becky754
1 Comments
Sorted by latest first Latest Oldest Best
If you don't want robots crawling those directories but don't want to announce them in your robots.txt file use the x-robots-tag HTTP header to block them.
Put this in a .htaccess file in any directory you don't want indexed:
Header set x-robots-tag: noindex
That will tell robots to ignore the contents of the files in that directory. This way no robots.txt file is necessary and you have your security through obscurity and a legitimate way of telling the search engines to stay out.
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.