: A robots.txt can disallow crawling, not indexing. The next time Google tries to crawl your pages, it will probably check your robots.txt and notice that they are no longer allowed to crawl.

@Shanna517

A robots.txt can disallow crawling, not indexing.

The next time Google tries to crawl your pages, it will probably check your robots.txt and notice that they are no longer allowed to crawl. This would stop Google from visiting your pages, but they don’t necessarily remove these pages from their index (nor does it mean that new pages won’t be indexed; they could find links to these pages somewhere else). Your pages could still be listed in their index (but without taking the title or the snippet from your page).

If you want to stop indexing, you’d need to use the meta-robots element or the X-Robots-Tag HTTP header. In that case, you’d have to allow crawling of these pages in robots.txt, otherwise Google would never be able to learn that you don’t allow indexing.

Vote Up Vote Down

Login to post a comment!