: John Conde already explained that your current solution won't work (+1), as Googlebot will only evaluate the rules after being redirected already and obey Disallow: /accordingly to block all

@Sims2060225

John Conde already explained that your current solution won't work (+1), as Googlebot will only evaluate the rules after being redirected already and obey Disallow: /accordingly to block all URLs including your root/main page thereafter.

However, for Googlebot specifically you can facilitate its extensions to the official robots.txt protocol (see section Pattern matching within Block or remove pages using a robots.txt file), like so:

User-agent: Googlebot
Allow: /$
Disallow: /

The $character is matching the end of a URL, so the Allow: /$ directive will allow any URL that begins with your domain name, followed by nothing, i.e. allow your root/main page after being redirected from index.php already.

The Disallow: / directive will block all other URLs as usual thereafter.

I've just tested this successfully with the robots.txt test tool in Google webmaster tools (available there via Crawler access under Site configuration), so you could confirm this yourself immediately likewise - looking into this invaluable selection of tools/information related to SEO in general and Google in particular is highly recommended anyway!

10% popularity Vote Up Vote Down