: Stop search engines from trying to crawl old sites on subdomains which are no longer available A few years ago I had couple of sites. Then I closed them. Back then, the search engines had
A few years ago I had couple of sites. Then I closed them. Back then, the search engines had added those sites of mine into their databases. Specifically Google and Chinese Baidu know about them. Now their bots query my DNS server for non-existent sites.
The complication is that now I've got a working website. Let's call it example.com. Those non-existent sites are subdomains like forum.example.com.
The traffic is not significant. But why to have it on a daily basis?
As far as I see it, there are two ways to possibly deal with it:
An administrative way, ie., to contact somehow the search engines and ask them nicely to remove those sites from their databases
A technical way, ie., to deal with it myself with some BIND9 configuration.
But I just don't know how to approach the issue, frankly.
More posts by @Murray155
4 Comments
Sorted by latest first Latest Oldest Best
Google never forgets a site that it had once indexed. Googlebot will continue to try to visit your old sites for years. There is no way that I know of to get the bots to totally stop trying to visit the old URLs.
If the sites are truly down, Google won't include them in the search index. You won't get real users visiting those sites from search engines. You will only see occasional hits from the search engine bots trying to see if your site is back up yet.
It sounds like you have token the DNS entries for the sub-domains away. The only activity you will see is occasional DNS queries for the subdomains. This is the least amount of activity from search engine bots that you can hope for.
If you want to serve an error page, use the "410 Gone" HTTP status for it. That status makes it explicit that you removed the content. With that status, search engine bots will come back less often than with other statuses such as "404 Not Found." (They will still come check occasionally, though.)
If you have access to those sites, add a hard 404-header. It won't get more obvious than that. The meta's (noindex and nofollow) still indicate that some content exists, you just don;t want it indexed (right now as far as they care).
If those sites dont exists, use a htaccess and some CNAMES to do that same trick. (CNAME forum.site.com as site, and htaccess it to 404).
You could set up a single page on the subdomain with a message to users that the subdomain is no longer active, and they should access example.com instead. Provide a link for users to click through to the main site, and include <meta name="robots" content="noindex"> in the header for search engines to know that it's no longer available.
This option is the most helpful to both users and search engines.
If you still have control on your websites before, just add this on the head
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
That will instruct the Search Engine spiders to stop indexing your website and stop following links. Those entries will be flushed from their database. Google on the other hand uses more than 1 database and it may take around 7 days before all copies of the database gets purged.
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.