: Does the Instant Preview in Google Webmaster Tools take robots.txt in account? Is that the way to go if I want to visually see what the googlebot sees? I'm trying to check a folder which I

@Harper822

I believe instant preview in Webmaster tools does take in account of robots.txt as the faqs for Instant Preview state that an error would be generated if blocked.


Roboted: The resource is being disallowed from being fetched through
a robots.txt directive for Googlebot.

sites.google.com/site/webmasterhelpforum/en/faq-instant-previews#14

Vote Up Vote Down

@Miguel251

If you followed support.google.com/webmasters/answer/156449?hl=en then you should be fine. The robot.txt should block those areas properly. Wait a couple of days to a week or so then check Webmaster Tools (sign up if you haven't already) and it will show you how many pages were blocked by robots.txt and you will see those pages vanish from Google Search with time.

Also if you have a sitemap that you submit to Google or even a HTML sitemap, don't have the blocked pages listed on there as this can lead to those pages being indexed even though they are blocked by robots.txt

Even if you block things by robots.txt it does not guarantee that search engines won't index it.

Vote Up Vote Down