: Question about duplicate groups (agents) in robots.txt I'm looking at a corner case with robots.txt, and am curious if you have any guidance. Basically, I'm in a situation where multiple people

@Shanna517

In the draft A Method for Web Robots Control from 1996 it says:


The robot must obey the first record in /robots.txt that contains a User-Agent line whose value contains the name token of the robot as a substring.
[…]
If no such record exists, it should obey the first record with a User-agent line with a "*" value, if present.


(Note that this is not the original robots.txt documentation, which would be www.robotstxt.org/orig.html from 1994, but it doesn't include info about this part.)

As you noted, also the Google documentation says:


Only one group of records is valid for a particular crawler.


So I'd say, strictly speaking only one block has to be taken into account. So yes, following blocks could be ignored.

Of course it might be true that some robots.txt parsers (including Google's) also add following record groups into account, but I think you shouldn't rely on it.

Vote Up Vote Down

@Frith620

Googlebot interprets it as one block. Meaning they will not crawl URLs that match any of the four rules.

This is super easy to find out since Google gives you a robot.txt testing tool once you are registered with Google Webmaster Tools. This is extremely useful for manyreasons, so I strongly recommend you do so. It would have at least avoided you waiting for this answer :)

Vote Up Vote Down