: How can I clone or mirror a site without SEO penalties for duplicate content? I am a web developer and I want to create clones of the sites I've developed for clients, so that I have an

@Angela700

robots.txt is not going to stop Google or others from Indexing the blocked pages. It's simply asking them to not crawl these folders.

If however your main domain, or a client links directly to the blocked pages Google will index the page.

Be sure to add noindex and nofollow to those pages meta tags, but to be safest you should password protect the folder they are in.

Vote Up Vote Down

@Heady270

The best solution is rel="canonical". Some robots are bad robots and will crawl your page, then they will place links on their SERP's and after that Google will know about them. I've tested this with one of my websites, and some of the links was indexed, even there are rule - User-agent: * Disallow: /

Good luck

Vote Up Vote Down

@Carla537

The problem with web bots is that you have to assume that they will follow all the rules that you've set out.

If they're going to follow any of them, it will be the robots.txt file, so having just that that should be enough. However, the rest won't hurt.

Vote Up Vote Down

@Kaufman445

Use robots.txt and put the following inside:

User-agent: *
Disallow: /


That's really all you need. Also, if a client's website is indexed as the original one, it won't get a penalty from duplicates.

Vote Up Vote Down