: Robots.txt file Question User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /account.php Disallow: /cart.php Is the above actually blocking crawlers? i just saw the extra "disallow" after

@Shanna517

Note that this is not a valid robots.txt. Parsers may understand it anyway, but you can’t be sure, especially not for the future.

Each record needs to be separated by at least one blank line.

So your robots.txt should look like:

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /account.php
Disallow: /cart.php


Meaning of this robots.txt:


the bot Mediapartners-Google may crawl all resources
all others bots may crawl all resources except those whose URL paths start with account.php resp. cart.php.

Vote Up Vote Down

@Nimeshi995

When you have Disallow followed by nothing, that effectively means "allow all":

Disallow:


If you want to block all, that's:

User-agent: *
Disallow: /


So, in your example, you are allowing Mediapartners-Google access to everything, and then disallowing all other crawlers from these two pages:

User-agent: *
Disallow: /account.php
Disallow: /cart.php

Vote Up Vote Down

@Sarah324

Actually, you have three lines here:


User-agent: Mediapartners-Google Disallow: User-agent: *
Disallow: /account.php
Disallow: /cart.php


You can get detailed info about the Google crawler from them

Vote Up Vote Down