: What is the objective of your robots.txt? Code like User-agent: * Disallow: means the whole site is crawlable for all search engines. So, what is the reason of adding User-agent: googlebot Allow:

@Jamie184

What is the objective of your robots.txt?

Code like

User-agent: *
Disallow:


means the whole site is crawlable for all search engines.

So, what is the reason of adding

User-agent: googlebot
Allow: /


?

P.S. The main objective of Robots.txt is not to open/close site resources for indexing, but to manipulate with "crawl-delay" option and "host" option (this is for Yandex Russia).

Vote Up Vote Down

Login to post a comment!