You can easily test in your Google Webmaster Tools account. But the answer is, no. It does not block your whole site. You block the /cgi-bin/ directory in line three. Line two essentially explicitly allows your entire site to be crawled. See the robots.txt documentation for more on this.

10% popularity Vote Up Vote Down

Feed

: Does the following robots.txt block all pages I want to knew if the following robots.txt file block the whole pages from being indexed or not. User-agent: * Disallow: Disallow: /cgi-bin/

More posts by @Welton855

: Can't Remove Page from Google High. My name is Festus. I'm creating a web site for a customer and would prefer not to have Google index the site while it's still being developed. The customer,

: Hreflang and no return tags with canonicals my question is in direct response to the following question... Canonical vs HREFLANG I am in a similar situation. However we are using canonicals in

: Can I have duplicate Microdata properties on a single page? I’m trying to work with Schema.org for the first time. Is it OK if I have duplicate itemprop attributes? Meaning, If I have my

: Want to redirect my site all pages to /blog For example my site: www.example.com When anyone visits my site, I want them all to redirect to www.example.com/blog But when I open my site it

Login to post a comment!

1 Comments

Back to top | Use Dark Theme