: Directing crawlers to content in language per language sub-domain I have a site with multilingual website with many pages (40M). The site has UGC, and each translation is actually for the titles.

@BetL925

Google says that you should avoid mixing two languages in one page. So you should not let Google index your pages that have translated titles, but original language body.

One way to do so would be to put all of the pages with the translated titles into a single subdirectory like:


en.example.com/translated/this-was-spanish.html
en.example.com/translated/this-was-german.html
es.example.com/translated/this-was-english.html
de.example.com/translated/this-was-english.html


then put Disallow: /translated/ into robots.txt. That would prevent Googlebot from both crawling and indexing the content that hasn't been fully translated.

If you can't move all the partially translated content to a single directory, then you could use the meta robots noindex tag on any page that has a translated title but an untranslated body. Google would still crawl these pages, but it would not include them in the index.

Vote Up Vote Down

@Mendez628

I do not recommend keeping same content in all sub-domain as it is against search engine's guidelines and may result in poor performance.

If you are operating different country, you can have same content but still you will have to change the offer price/services location etc. To redirecting crawlers is not a good practice as there is not country specific crawlers. Search engine may rank page based on relevancy and 200+ other factors.

Vote Up Vote Down