: How to allow Googlebot and Bingbot to crawl mysite? Recently I got email message from Google that Googlebot cant access your site. I tried to look for some articles on the internet about this

@Angela700

Based on the contents of your robots.txt file and your needs, you're better off making that file entirely empty or even better, maybe with a link to the sitemap files so search engines can find your URLs.

I suggest this because crawlers normally think they are allowed to access anything. By specifically using Disallow, you're trying to tell at least one crawler that they are not allowed to access something on your site.

Here's a link to an example of a robots.txt that google uses:
www.google.ca/robots.txt

Vote Up Vote Down

@Pope3001725

No, the last two lines of your robots.txt file take precedence over the first four as the syntax of those first four lines is incorrect. As a result Google is blocked from viewing your website.

To allow Google and Bing you must specifically and individually allow each crawler:

User-agent: googlebot
Disallow:

User-agent: bingbot
Disallow:

User-agent: *
Disallow: /


Going forward, to find out if your robots.txt file will allow Google to index your site test it inside of your Google Webmaster Tools account. Also see the official robots.txt documentation.

Vote Up Vote Down