: Google claims not to find my robots.txt Like I said in the title, the Google Webmaster Console claims not to find the robots.txt file of my homepage and therefore it won't crawl it. But there

@Odierno851

If you're seeing this in the "crawl errors" section as "Inaccessible robots.txt" then that would be due to network or server issues when Google tried to fetch the robots.txt file (eg, server returned a 500 or timed out for the robots.txt). You can check to see if Google has the current version by using the "robots.txt test tool" in Search Console -- that should show the current contents of the file, and the dropdown on top shows you the difference over time.

Vote Up Vote Down

@Deb1703797

As much as we all think google is smart at processing everything in a perfect manner, I believe there are still some things it can't do, and I think one of them is establishing a secure connection just to download a plain text file.

What I would suggest trying in order to make every single search engine happy is to make your robots.txt accessible without requiring a secure connection to be established first.

When I tried accessing:
soundhaus-band.de/robots.txt

I get a redirect to:
soundhaus-band.de/robots.txt

This can throw search engines off because they're looking for robots.txt content only to find a redirect.

You should make soundhaus-band.de/robots.txt produce the exact same output as soundhaus-band.de/robots.txt to make everyone happy.

If that doesn't work, then send google negative feedback about their robots.txt detection service.

Vote Up Vote Down