: Crawl Error on Disallowed Content I've got a rule in my robots.txt file: # Crawlers Setup User-agent: * # Directories Disallow: /my_directory/ Yet I am getting entries in my Crawl Errors section

@Lee4591628

Robots.txt does not block search engines from indexing a link, only the content of that link. So what is likely happening is that Googlebot is visiting your page even though it knows it can't index the contents of that page, only the url.

From Google Webmasters:


Blocking Google from crawling a page is likely to decrease that page's ranking or cause it to drop out altogether over time. It may also reduce the amount of detail provided to users in the text below the search result. This is because without the page's content, the search engine has much less information to work with.

However, robots.txt Disallow does not guarantee that a page will not appear in results: Google may still decide, based on external information such as incoming links, that it is relevant. If you wish to explicitly block a page from being indexed, you should instead use the noindex robots meta tag or X-Robots-Tag HTTP header. In this case, you should not disallow the page in robots.txt, because the page must be crawled in order for the tag to be seen and obeyed.

Vote Up Vote Down

@Sherry384

Robots.txt blocks only content, not URL. So, if server response on the "blocked" URL is wrong, Google can get it.

Vote Up Vote Down