: Dev site indexed by Google I was developing a new version of our site on a subdomain (dev.oursite.com) and the robots.txt with the Disallow got replaced at some point, so the site got indexed

@Angela700

I'm always concerned about development sites becoming indexed. I don't trust robots.txt or meta noindex, while I do use them I also password protect the sites if it's not to inconvenient. But another option is to use .htaccess and deny access to everyone except IP's within your company and for co-workers and developers. Just add their class C.

Vote Up Vote Down

@Miguel251

In addition to the correct answer provided by Joshak I would like to provide you with a tip on how this can be prevented.

What I have done to solve this exact same issue is to force the robot.txt in the Apache httpd definition of the vhost. This way there is no way the "disallow" can vanish or be changed by any code in the website under development.
My vhost definitions all look something like this:

<VirtualHost *:80>
DocumentRoot /var/www/html/Hosting/test.example.com/newsite.com
ServerName newsite.com.test.example.com
ServerAlias *.newsite.com.test.example.com
UseCanonicalName on
RewriteEngine on
RewriteRule ^/robots.txt /var/www/no-indexing-robots.txt [NC,L]
</VirtualHost>

Vote Up Vote Down

@Welton855

Check out the URL removal tool in Google Webmaster Tools. I'd also 404 the pages instead of redirecting them to get them removed faster, in the future beyond robots.txt you could drop in the rel="canonical" to make sure Google knows the dev site is just a copy of the main site and is not to be indexed.

Vote Up Vote Down