: Important update Google may not respect your robots.txt directives, as stated here: https://developers.google.com/webmasters/control-crawl-index/docs/faq#h17 However, robots.txt Disallow does not guarantee

@Angela700

Important update

Google may not respect your robots.txt directives, as stated here: developers.google.com/webmasters/control-crawl-index/docs/faq#h17

However, robots.txt Disallow does not guarantee that a page will not
appear in results: Google may still decide, based on external
information such as incoming links, that it is relevant. If you wish
to explicitly block a page from being indexed, you should instead use
the noindex robots meta tag or X-Robots-Tag HTTP header. In this case,
you should not disallow the page in robots.txt, because the page must
be crawled in order for the tag to be seen and obeyed.


Don't know when Google changed this, but is how it works right now.

Vote Up Vote Down

Login to post a comment!