: Robots.txt on one line Can I combine robots.txt rules on one line, separated by whitespace? Will this work? User-agent: * Disallow: /

@Speyer207

The file MUST consist of one or more records separated by one or more blank lines.

Robots.txt must use this format:

<field>:<optionalspace><value><optionalspace>
<field>:<optionalspace><value><optionalspace>


Which looks like:

User-agent: *
Disallow: /


Or without optional spaces and more line breaks:

User-agent:*

Disallow:/


Useful Links


A Standard for Robot Exclusion
Robots Internet Draft
Robots.txt Specifications

Vote Up Vote Down

@Gretchen104

No. See the Google Robots.txt specifications, specifically the File Formats section.


The expected file format is plain text encoded in UTF-8. The file consists of records (lines) separated by CR, CR/LF or LF.

Only valid records will be considered; all other content will be ignored. For example, if the resulting document is a HTML page, only valid text lines will be taken into account, the rest will be discarded without warning or error.


So line breaks between lines are necessary.

Vote Up Vote Down