: With Robots.txt disallow all, why was my site still getting traffic? In failing to check my robots.txt on a community site, it had: User-agent: * Disallow: / I only clued into this because

@Sent6035632

Add <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">in your website's index page to stop indexing.

Vote Up Vote Down

@XinRu657

Using robots.txt is the first step for remove your website from Google search results.
You should use "Remove URL Tool" from your Google Webmaster Tools account to tell Google remove your pages or completely your website from its search results.

Vote Up Vote Down

@Kristi941

Robots.txt just tells search engines not to crawl your pages. It does not tell them to not index your pages. So if your pages have links from other websites the search engines will know they exist. And because off-page factors affect rank, sometimes greatly, your pages can rank well for long-tail search terms without ever being crawled.

To actually prevent search engines from actually indexing your pages you will need to use the x-robots-tag.

Vote Up Vote Down

@Bryan171

You just indicated to search engines to not crawl your webpages, not to desindex them. That's why, URLs are still indexed and you get traffic from them.

Vote Up Vote Down

@LarsenBagley505

Robots.txt doesn't stop indexing, it only stop crawling. So your pages could still be indexed and traffic referred with a disallow : / directive. Though as Google wouldnt be able to crawl your site then you could expect for traffic to diminish in time.

Vote Up Vote Down