: Is it OK for SEO and ads on the site if I have the same content on multiple domains? I have a web based proxy and web filters eventually blacklist the domain. Because of this, I am thinking

@Eichhorn148

You don't want to block duplicate content with robots.txt. Google's guidelines say:


Google does not recommend blocking crawler access to duplicate content on your website, whether with a robots.txt file or other methods. If search engines can't crawl pages with duplicate content, they can't automatically detect that these URLs point to the same content and will therefore effectively have to treat them as separate, unique pages. A better solution is to allow search engines to crawl these URLs, but mark them as duplicates by using the rel="canonical" link element, the URL parameter handling tool, or 301 redirects.


The rel canonical link tag is what you want to use. It is for cases where users need to be able to use various URLs but where search engines need to know which one in the correct one to index. Every page on your site should get a tag in the head like:

<link rel="canonical" href="http://mainsite.example.com/this-page.html" />


As for Ads, the AdSense TOS says:


Publishers are not allowed to create multiple pages, sub-domains, or domains with substantially duplicate content.


The "duplicate content" is linked to the document that says that you can use rel canonical links to solve the problem. So implement the rel canonical links and you should be fine for both search and AdSense.

Vote Up Vote Down

@Harper822

If you prevent robots from indexing content, these websites are not considered as duplicate versions, and Google will ignore them.

In fact, noindex is usually applied to "print version" of web articles. That's the most similar scenario. In other scenarios, I would suggest to use redirection or canonicalization.

For the ads: it depends. You need to read the ToS and see if some sort of limitation is present.

Vote Up Vote Down