: Can creating a duplicate site for testing and upgradation can cause duplicate content Considering upgrade of CMS to latest version- the process may take anywhere from 2 to 4 weeks. May suggest

@Cofer257

John is correct about adding authentication. One alternative is to restrict the site to specific IPs (e.g. the IP of your office), and serve a 404 or 403 error to others.

Also, with regards to robots.txt, you should block all bots, which is much simpler:

User-agent: *
Disallow: /


Even if you only block via robots.txt, there is no downside for SEO. The test site will not be crawled. If any individual page has a link to it from elsewhere that page may be crawled but never the whole site.

Vote Up Vote Down

@Correia994

You are describing a staging site. As you suggest, you will replicate the site and its production environment locally and to a staging site for development and testing.

Also, as you have mentioned, you may restrict access to the staging site by filtering by UserAgent and also with robots.txt. I recommend blocking access to this staging site using an Authentication method, such as Basic Auth as mentioned by OP.

You may wish to research a Development -> Staging -> Production workflow. Sounds like what you are trying to achieve.

Vote Up Vote Down

@Pope3001725

You must make sure that this site is unavailable to search engines. The best way to do this is to put the entire dev site behind a login mechanism. Basic Authentication is the simplest and most effective way to do this. It makes it impossible for search engines to crawl but is easy for developers/users to access as they only need to enter a password once per visit.

Vote Up Vote Down