: Is it safe to block redirected (but still linked) URLs with robots.txt? I have a website that has all URLs optimized and 301 redirected from nasty URLs to clean ones. However, everywhere throughout

@Ogunnowo487

If you disallow the unclean URLs in robots.txt, polite bots will no longer visit these URLs. So they will never notice that you 301-redirect them to other URLs that they’d be allowed to crawl. Bots that don’t know your clean URLs yet would only be able to visit these pages when they are directly linked to with the clean variant (not the blocked unclean variant).

So you should not block them in robots.txt.

As you are 301-redirecting your unclean URLs to corresponding clean URLs, you don’t need to do anything. Bots will know what to do. If some search engines still have some unclean URLs indexed, it should only be a matter of time until they update their index.

Vote Up Vote Down

@Kaufman445

The redirection won't work on those blocked URLs. After adding 301 redirection you will not need to apply canonical as well. This won't effect the websites I have seen big brands changing the URL and make millions of redirection just like the SEOmoz did after changing its name to Moz.

Vote Up Vote Down

@Speyer207

There's no to need to block anything in robots.txt these days, simple use rel="canonical" on your pages and you never risk duplicate pages regardless if its accessible via unclean or clean because the canonical will tell Google the preferred.

In regards of the indexed non-clean URLs if you use canonical's these will deindex/update themselves however if you don't have page replacements and you just want to remove them then using both noindex in your head HTML and disallow: /unclean-url/ in your robots.txt (Google recommends both robots.txt and noindex usage).

Also for SEO purposes you should correct those URLs as soon as possible as your losing page juice due through the 301 redirect.

Vote Up Vote Down