: Keep in mind that the robots.txt controls crawling, not indexing (docs). It's possible that a URL is indexed even if it has never been crawled before. In general, that doesn't matter -- it's

Keep in mind that the robots.txt controls crawling, not indexing (docs). It's possible that a URL is indexed even if it has never been crawled before. In general, that doesn't matter -- it's not like the URL is going to show up in search results if the rest of your site is indexed normally. If you do want to be certain that these URLs do not show up in search results, you can either:

Allow crawling (remove the disallow) and serve a "noindex" robots meta tag with the pages
Use the URL removal tools in Google Webmaster Tools to have those URLs removed from Google's index

FWIW I would also reconsider disallowing CSS/JavaScript, as this can be used to generate preview images for your pages. Also, don't use robots.txt as a means of canonicalization (the ".php" and "?" in your robots.txt file), if we can't crawl it, we can't recognize that you have a better version.

10% popularity Vote Up Vote Down