: Does Google respect a redirect header for robots.txt to a different file name? With respect to robots.txt requests, Google states "Redirects will generally be followed until a valid result can
With respect to robots.txt requests, Google states "Redirects will generally be followed until a valid result can be found".
Does this mean that, if I create a 301 redirect for www.example.com/robots.txt to www.example.com/myrobots.txt, Google would parse the content served from www.example.com/myrobots.txt as being valid robots.txt content? Or does the fact that the result URL is not named robots.txt make this an invalid result, and therefore will be ignored?
More posts by @Kaufman445
1 Comments
Sorted by latest first Latest Oldest Best
Robots follow the Robots exclusion standard, also known as the Robots Exclusion Protocol. According to the WC3, robot directives must be specified in a file with the URI containing "/robots.txt", all in a lower case string, located in the root directory of your site. For more on that see: WC3 - The robots.txt file
As stated in there: The Robot will simply look for a "/robots.txt" URI on your site.
Therefore, some robots may not recognize a robots.txt with a different file name or location.
In Google Webmaster Tools, you can view the URL to the robots.txt file for your site under Crawl->Blocked URLs, as covered here: Google Webmaster Tools - Block or remove pages using a robots.txt file (under the "Test a robots.txt file" down arrow).
If your robots.txt is missing, it likely did not find or recognize it. You can reconfirm this by listing your URL to test against and click "Test".
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.