: I would take advantage of Google's regular expression support. You want to allow /tagA1 and /tagA1/, but not /tagA11/XXX. So: User-agent: * Allow: /tagA1$ Allow: /tagA1/$ Disallow: /tagA1 The $

@Angie530

I would take advantage of Google's regular expression support. You want to allow /tagA1 and /tagA1/, but not /tagA11/XXX. So:

User-agent: *
Allow: /tagA1$
Allow: /tagA1/$
Disallow: /tagA1


The $ denotes end of string. So /tagA1 will be allowed, but tagA1/foo will be blocked.

Note that Google and Bing support the $. Others might not. Note also that the order of Allow and Disallow here is important. Many crawlers read the robots.txt sequentially and apply the "first matching rule". If you put the Disallow first, those crawlers would never index the pages you allow.

Vote Up Vote Down

Login to post a comment!