: How do I disallow variable parent directories and subdirectory with robots.txt? I want to disallow robots to index some pages of my site. For example, I want to disallow the following pages:

@Ogunnowo487

To disallow crawling of the URLs ending in page/ then yes, you can use a wildcard like you mentioned. However, there is no need to use a wildcard at the end of the pattern, since robots.txt is prefixing matching by default.

User-agent: *
Disallow: /category/*/page/


That will still allow the crawling of the less specific URLs mentioned.

Note that the wildcard * is an extension of the "standard". Whilst all the main search engine bots support it, not all (good) bots do.

Vote Up Vote Down