: Robots.txt and Regular Expressions If I place a regular expression in robots.txt will it be readable for the web crawlers?

@Heady270

While it's not part of the standard robots.txt protocol (and therefore not globally recognized), Google and Bing both support LIMITED pattern matching.

For example:

User-agent: Googlebot
Disallow: /private*/


Will block Googlebot from any directory beginning with "private". Currently, Googlebot supports * and $ (end of string).

Details on Googlebot's pattern matching: www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

Vote Up Vote Down

@Margaret670

No, the robots.txt syntax only supports wildcards, and not regular expressions. See here for a good discussion of the syntax: www.robotstxt.org/robotstxt.html.

Vote Up Vote Down