: This is my experience with crawl errol by robots.txt. A few months ago i've found the message like "URL restricted by robots.txt", my site is Blogger (*.blogspot.com). I ask my problem to

This is my experience with crawl errol by robots.txt. A few months ago i've found the message like "URL restricted by robots.txt", my site is Blogger (*.blogspot.com). I ask my problem to Google Webmaster and they send me the answer about my robots txt as you can read :

rayhanzhampiet.blogspot.com :

"Google was unable to crawl the URL due to a robots.txt restriction: your robots.txt file might prohibit the Googlebot entirely; it might prohibit access to the directory in which this URL is located; or it might prohibit access to the URL specifically. Often, this is not an error"
can anybody tell me how can i cancel the robots.txt restriction?

Top Contributor Webmaster Help :

Hey rayhanzhampiet,
i guess we are dealing with your blog on blogger*com (blogspot*com)? If so you don't need to do anything about robots.txt restrictions.,

This is your robots.txt:
User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search -- si only restricts duplicated stuff that not even you want to have indexed
keep cool, see all URLs restricted have /search/ in their path as specified in your robots.txt?

They are restricted because they duplicate what is already seen in posts and on the homepage, for example here:
rayhanzhampiet.blogspot.com/search/label/Tips-Blogging You recognize having seen all this before --> on your original posts which are not restricted and thus indexed ;-)

/search/ has to be restricted to avoid duplication in the index, and, as I said, you even want to restrict them for robots for your own sake. This restriction is done by default on blogspot*com (and domains that use blogger like yours, of course) and does not need to be resolved in any way, it's not an error (messages in webmastertools are diagnostic only here).

I hope the people (blogger) who have a problem with robots.txt like me will known that they dont need to be resolved.

Thanks to Pro Webmasters,