: Robots.txt Blocking Blog Tags? Tags are causing duplicate content on my website and I want to make sure that I block them from being indexed. Is this the right syntax to use for blocking those

@Bryan171

I usually just add a noindex meta tag to tag pages. But your method in robots.txt should work as well, the only downside being that you'll need to manually add each tag, unless they're in a sub-folder of their own ie /blog/tags/tag in which case you could simply add Disallow: /blog/tags/*.

Vote Up Vote Down

@Nimeshi995

2009 - Google, Bing, and Yahoo announce Support for Canonical Links

There is no need to use robots.txt to tackle the duplicate content problem, in February 2009, Google, Yahoo and Microsoft (for their Bing search engine) announced support for the canonical link element which aims to tackle the problem of duplicate content.

Rel="canonical" To The Rescue

So rather than blocking Google and other search engines indexing the pages into results you should opt to use Canonicalization by using <link rel="canonical" href="http://www.yourdomain.com/path-here" /> This will tell Google and the other search engines which are the master pages and those that are duplicate and meaning that you will receive no slaps from Google.

SEO Plugins Make Canonicalization Easy

There are many plugins for various content management systems that automate this process for you, for example and assuming that you are using WordPress you can install Yoast SEO which will ensure that all pages use canonical and pages such as Tags will never harm your actual pages. I've used many CMS engines so if you using Drupal, Joomla, Blogengine or any other let me know and I'll recommend some alternatives.

References:

I recommend that you read more on canonical to give you a clearer indication what it can do for you:


Pro Webmasters How to use rel='canonical' properly
SEOMoz Canonicalization
Google Webmasters About Rel="Canonical"
Wikipedia Canonical Link Element

Vote Up Vote Down