: How to tell search engines to lower the robots.txt scanning rate on obsolete subdomains? After putting my site into one domain instead of multiple subdomains, I checked my logs and noticed that

@Odierno851

May be Google check your robots.txt frequently, because you have moved your site content to new location. There is no any way to control robots.txt crawling rate.

But, If your robots.txt did not blocked anything like any directory, then don't use robots.txt at all. For blocking other bots, you can use httacess. I have seen most of sites, don't use robots.txt properly. They specify useragent and allow them to crawl, like this,

User-agent: Googlebot
Allow: /
User-agent: Mediapartners-Google
Disallow:


This type of robots.txt is junk, it just waste space and bandwidth.

So Use Robots.txt only, if you really want to block such a thing, For small type of things, I recommended to use X-Robots header tag.

Vote Up Vote Down

@Bryan171

You should keep the subdomains setup with redirects to your top level domain - that way old traffic will land on a functional website and Google will re-crawl over time and realised all the traffic has been directed elsewhere

Add this to .htaccess and modify it to suite your sites Url

RewriteEngine on
RewriteCond %{HTTP_HOST} ^m.example.com$
RewriteRule ^ example.com/m%{REQUEST_URI} [L,P]


Read more here: stackoverflow.com/questions/1456956/redirect-subdomain-into-top-level-domains-folder

Vote Up Vote Down