: How to temporarily remove a website from search engines? We are working on a new website and we forgot to disallow robots after putting the test version online, on a subdomain. Now we are

@RJPawlick198

If you don't want the site crawled at all, I'd suggest moving it to a "dummy" subdomain, and be sure to update the rules to NOT crawl it before putting the code there (and ideally ensure there are no links from the "live" site to the "demo" site.

e.g.

a347demo.example.com

Vote Up Vote Down

@Pope3001725

is it ok if we disallow robots at this level without having any effect on the live website (same domain) when switching from allow all to disallow all?


Yes. Once you stop blocking the contents of the website it will be crawled again. But I would resubmit my site to Google and/or submit a XML sitemap just to make sure they know you now have content to index.


can we use the 307 http status (Temporary Redirect)?


That would work as well.


how long doest it take spiders to refresh their db (either disallow robots or 307 redirect)?


It varies. Newer sites will find they are crawled less frequently and changes made to the website are picked up slowly. I definitely wouldn't expect it to happen quickly. You'll find in questions asked here that it can take weeks or months to finally see results.

Vote Up Vote Down