: Removing full site from Google index I want to remove the contents from Google index. Google index include my websites huge number of pages indexed in Google index about 5,000,000 Pages earlier

@Samaraweera270

.htaccess




I just spent the last minute clicking around your site with my User-Agent set to Googlebot 2.1, and I didn't hit a single 410. I'm not an expert on .htaccess, but are you sure your .htaccess rule is functioning correctly site-wide?
F should produce a 403 (Forbidden), not the 410 your Fetch as Googlebot produced.
Why only tell Google the page is forbidden, gone, or whatever? Your meta noindex would suggest you wish to instruct search engines other than Google.



Meta Noindex


You seem to be instructing specific robots to noindex, and then cancelling that out by telling all robots to index:

<meta name="googlebot" content="noindex,nofollow">
<meta name="searchbot" content="noindex,nofollow">
<meta name="baidu" content="noindex,nofollow">
<meta name="geo.country" content="IN">
<meta name="robots" content="Index, Follow">




Robots.txt


Your robots.txt file does not, in fact, contain

User-Agent: *

Disallow: /


as you say it does. It contains

User-agent: *
Disallow: /judgment_view
Disallow: /payment
Disallow: /include
Disallow: /search.php*
Disallow: /admin


Although it isn't that important as it would only prevent crawling, and wouldn't remove content from the index.


Solution


You haven't stated exactly what your aim is, neither is it clear from the steps you've taken on your site, but the above should serve as a starting point.

Vote Up Vote Down

@Nimeshi995

Google doesn't drop of pages that quickly partly because pages are ranked and if they was to drop them quickly people would whine about them losing their page ranks with a unoticed mistake. So its kinda like a grace period to short things out.

The problem with .htaccess redirects is that Google can assume that it's a mistake within the HTACCESS so periodically it will come back and check again, and if you have many many pages then this because a time consuming process.

Additional you should do a on every page, because the robots.txt is not always checked on crawls, personally I wouldn't use .htaccess at all because they will just keep coming back and assuming its an error. NOINDEX is faster than robots and htaccess but try the removal tool I linked above.

<meta name="robots" content="noindex,nofollow">


Also another factor is how Google treats your site in terms of ranking and what it considers its speed. So if your on a VPS increase the speed of it, it should increase the amount of pages that Google will crawl due to the fact Google bot will crawl for a X amount of alloted time and then leave regardless, and you want as much juice as possible.

Best Method, Hit or Miss

Now the best way to remove URLS promptly is via their webmaster tools however you have 3 million pages that becomes unrealistically impossible however there is a site removal tool which many don't know about and funny enough the url is almost the same.

CHECK
www.google.com/webmasters/tools/removals www.google.com/webmasters/tools/url-removal?hl=en&siteUrl= (This one is the Webmaster Tools one - the one above you can request removals of sites)

Vote Up Vote Down

@BetL925

Would adding: <meta name="robots" content="noindex,nofollow"> to the head section of your site's pages help speed things along?

Vote Up Vote Down