: Robots.txt never downloaded but some blocked URLs in GWT There is something I don't understand in Google Webmaster Tools (GWT) for my WordPress site. In menu "Blocked URLs", it mention that my

@BetL925

Problem comes from encoding of the robots.txt file. UTF-8 encoding is recommended by Google.

Vote Up Vote Down

@Samaraweera270

It may just be a glitch – they do happen from time to time.

One other possibility occurs though: your screenshot is from GWT on a non-www version of your domain. Do you have the www version authorised as well (e.g., so that you can set preferred domain)? If so, and the www version is canonical, it could explain why Google would know how many URLs are blocked without technically having downloaded the file (they downloaded the www version… which is the same file really, but isn't in a pedantic techie sense).

Bit of a long shot, but that's about the only explanation, other than a glitch, I can come up with.

Vote Up Vote Down

@Annie201

Google will sometimes say it's been blocked by robots.txt when in fact it is a noindex tag preventing Google from indexing the page. If you have any plugins for maintenance mode or others which can possibly set noindex on your home page or other pages I would suggest you go to the page view the source in your browser and check that there is no noindex being rendered by a plugin.

Vote Up Vote Down