Robots that do not recognize wildcards (which is not in the official spec) will treat * as a literal character. The fact that it is not a valid URL character may mean that they ignore the rule altogether. In either case, it likely means that the rule will have no effect on them.

This will depend a bit on the exact implementation of the crawlers robot.txt honoring scheme and can not be entirely counted on.

If you want to avoid this you could have a separate configuration for googlebot (and others who do honor robots.txt.

E.g.

User-agent: *
Disallow: /

User-Agent: Googlebot
Disallow: /*action=*$

Which bans all robots except Googlebot which will honor the wildcard configuration.

10% popularity Vote Up Vote Down

Feed

: Robots.txt and pattern matching Adding this to my robots.txt User-agent: * Disallow: /action=$ How does robots not recognizing wild cards handle this?

More posts by @Bryan171

: There are lots plugins available in wordpress, joomla and drupal but i don't think it covers all of your needs. you want customize hotel booking system, as you said. I suggest you to make

: Privacy policy and terms of use language I have a Czech registered business with which I'm serving a web app mostly (but not exclusively) targeted to Italian customers. The server is in Amsterdam.

: Webmaster tools 500 crawl error for asp faceted navigation that does not exist I am getting around 2,500 type 500 URL errors in Google webmaster tools. These pages are faceted navigation results

: Need to redirect to another server with no access to DNS A record I've a website which is on two servers - the one it's on now malfunctioned. I have the site hosted on a second server,

Login to post a comment!

1 Comments

Back to top | Use Dark Theme

: Robots.txt and pattern matching Adding this to my robots.txt User-agent: * Disallow: /*action=*$ How does robots not recognizing wild cards handle this?