: How to resolve "Google can't find your site's robots.txt" error? I've recently found that "Google can't find your site's robots.txt" in crawl errors. When I tried Fetching as Google, I got result

@Hamaas447

If you have a lot of websites stored on the same server, and use the same Google Analytics account for them all, and also use a third party hosting service such as GoDaddy, NetWork Solutions, etc... with a dedicated server, the hosting company may be blocking the Googlebot after it tries a certain number of the websites that you have hosted.

The Googlebot and the Fetch as Google tool run on two different IP numbers, which would account for the ability for you to Fetch as Google successfully, but that the Googlebot could not find your robots.txt.

Vote Up Vote Down

@Ann8826881

Side note: Your robots.txt syntax is not correct:

User-agent:

*Disallow: /wp-admin/

Disallow: /wp-includes/



I guess the * moved to the next line accidently; it should be User-agent: *
*Disallow will be unknown to most parsers and therefor the followed rule will be ignored.
no line breaks allowed inside of a block


So it should be:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

Vote Up Vote Down

@Shelley277

Make sure that in Settings → Privacy in the back-end you've checked Allow search engines to index this site and not the other one which is Ask search engines not to index this site.

Also, that was probably just a typo, but your robots.txt should look like this:

User-agent: *

Disallow: /wp-admin/
Disallow: /wp-includes/

Vote Up Vote Down