: Can Googlebot handle robots.txt with a 302 redirect? In Google Webmaster Tools, I get an error: Google couldn't crawl your site because we were unable to access your site's robots.txt
In Google Webmaster Tools, I get an error:
Google couldn't crawl your site because we were unable to access your site's robots.txt file
The associated help says:
If your robots.txt file exists but is unreachable (in other words, if it doesn’t return a 200 or 404 HTTP status code), we’ll postpone our crawl rather than risk crawling disallowed URLs
My site 302 redirects all http traffic to HTTPS - so access to blah/robots.txt is 302 redirected to blah/robots.txt, so literally it doesn't return 200 or 404 as requested above.
My question - does Googlebot object to the 302 redirect when it attempts to access the robots.txt file?
Note: a lot of the server configuration is out of my control, and is configured this way due to corporate IT. I'm just the poor guy who needs to get this working in spite of the constraints.
More posts by @Deb1703797
2 Comments
Sorted by latest first Latest Oldest Best
These quotes I have taken from Google.
File location & range of validity
The robots.txt file must be in the top-level directory of the host, accessible though the appropriate protocol and port number. Generally accepted protocols for robots.txt (and crawling of websites) are "http" and "https". On http and https, the robots.txt file is fetched using a HTTP non-conditional GET request.
and also
3xx (redirection)
Redirects will generally be followed until a valid result can be found (or a loop is recognized). We will follow a limited number of redirect hops (RFC 1945 for HTTP/1.0 allows up to 5 hops) and then stop and treat it as a 404.
You could/should maybe read the source at Google.
Most likely yes, as you are using a 302 redirect which is temporary redirect that Google recommends against using. You should be using a permanent 301 redirect to redirect from HTTP to HTTPS.
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.