: Second form do nothing, it's just full-line comment The file consists of one or more records separated by one or more blank lines (terminated by CR,CR/NL, or NL). Each record contains

@Turnbaugh106

Second form do nothing, it's just full-line comment


The file consists of one or more records separated by one or more
blank lines (terminated by CR,CR/NL, or NL). Each record contains
lines of the form ":". The
field name is case insensitive.

Comments can be included in file using UNIX bourne shell conventions: the '#' character is used to indicate that preceding
space (if any) and the remainder of the line up to the line
termination is discarded. Lines containing only a comment are
discarded completely, and therefore do not indicate a record boundary.


User-agent: Slurp Disallow: /


works the same way as multiline definition


It seems like for someone disallowing a large number of crawlers at / then a single line notation would be easier to read


It depends. You can use multiline definition and multi-Useragent per URL because


If more than one User-agent field is present the record describes an
identical access policy for more than one robot. At least one field
needs to be present per record.


of write oneliner for each crawler

Vote Up Vote Down

Login to post a comment!