: Robots: block /lang/page from the index but keep /page My URL structure looks a bit like this: /group1/ /group2/ /group3/ Group3 (and it's subpages) are also available in another language at

@Ogunnowo487

If you literally only have a few "groups" you want to block then you would do something like:

User-agent: *
Disallow: /lang/group1
Disallow: /lang/group2


...and everything else would be allowed. This would work with all robots that obey the original "standard". Or, you could block all groups (group1, group2, etc.) and make an exception for "group3", like:

User-agent: *
Disallow: /lang/group
Allow: /lang/group3


Note that the Allow directive is not part of the original "standard", but has universal support. The URL path is simply a prefix.

HOWEVER, I wouldn't use robots.txt to block the pages being "crawled". What about stray visitors? And bad bots? And robots.txt doesn't prevent pages from being indexed if they are inadvertently linked to. I would use .htaccess or your server config to actually block all traffic to these URLs. Something like the following in .htaccess:

RewriteEngine On
RewriteRule ^lang/group[12] - [R=404]


To respond with a 404 for all requests to these invalid URLs. Or, use the F flag to respond with a 403 "Forbidden".

Vote Up Vote Down