: Googlebot is blocked from my domain My robots.txt is: User-agent: * Disallow: / Sitemap: http://example.com/sitemap.xml Sitemap: http://example.com/sitemap.xml.gz It has been the same for well over

@Ann8826881

Your robots.txt tells all bots (User-agent: *) not (Disallow:) to crawl any URL starting with a slash (/), which would be all URLs, of course.

If you want to allow all bots crawling all your pages, remove the slash:

User-agent: *
Disallow:


Note that Allow: is not part of the original robots.txt specification, however, some/many crawlers recognize it. Disallow: and Allow: / would be equivalent in this case, so why not use the "standard" one.

Vote Up Vote Down

@Bryan171

Your robots.txt own these lines:

User-agent: *
Disallow: /


It indicates to Googlebot and other search spiders to not index your website.

Just put these lines instead and wait:

User-agent: *
Allow: /


That way, all search spiders (Googlebot included) will index your website.

Vote Up Vote Down

@Turnbaugh106

I can confirm that your robots.txt and HTML are not blocking Google from scanning your site, so whats the problem? In short... time.

Google may state that it takes two days for Google to take note of a robots.txt but in no way does this imply that all these pages blocked previously will be unblocked and instantly crawled.

Now that Google can crawl your site it may take a few visits to gather the data to index into their engine. This varies and can be one to two weeks, but sometimes sadly it does take longer. You just need to be patient nothing else you do in Google Webmaster Tools will speedup this process.

It's also important to note that even if Google visits your site say for example on the front page it doesn't necessary capture all the data they may want, sometimes it takes a few crawls. Again, just sit back and get some lemonade. ;)

Vote Up Vote Down