: Robots.txt: block all webpages except a few number of webpages? I have a few doubts regarding robots.txt. Say, my domain is stackoverflow.com, A) Will the code below do the following for all

@Ogunnowo487

Your robots.txt is invalid. Line breaks are not allowed in a record. So it should look like:

User-agent: *
Disallow: /
Allow: /$
Allow: /a/$
Allow: /a/login.php
Allow: /a/login.php?return=/pligg/



Will the code below do the following for all the crawlers?


No, your robots.txt won’t work that way for all crawlers.

Allow is not part of the original robots.txt specification. Only some parsers will understand it (and they might have implemented the wildcards differently), all other parsers will ignore the Allow lines.

Vote Up Vote Down

@BetL925

A) Yes for the most important ones (Googlebot...).


Yes


B) The right file name is robots.txt.

Otherwise, the right code to do what you want is the following:

User-agent: *

Disallow: /

Allow: /a/$
Allow: /a/login.php
Allow: /a/login.php?return=/pligg/

Vote Up Vote Down