: Is my whole site is blocked by this Disallow: / This is a view of my robots.txt file did this query blocked only Baiduspider Search Engine or all of the search engines to check my site.

@Ann8826881

The Baiduspider bot is not allowed to crawl anything on your host.

All other bots are not allowed to crawl any URL whose path begins with /templates_c/. They may crawl all other URLs on your host.

Vote Up Vote Down

@Tiffany637

If you are using disallow:/ with user-agent: * then only entire site is blocked with robots.txt.

Otherwise only particular user-agent will get blocked with your sites.

Blocking user-agents:

The Disallow line lists the pages you want to block. You can list a specific URL or a pattern. The entry should begin with a forward slash (/).

To block the entire site, use a forward slash.

Disallow: /

To block a directory and everything in it, follow the directory name with a forward slash.

Disallow: /junk-directory/

To block a page, list the page.

Disallow: /private_file.html

To remove a specific image from Google Images, add the following:

User-agent: Googlebot-Image
Disallow: /images/dogs.jpg

To remove all images on your site from Google Images:

User-agent: Googlebot-Image
Disallow: /

To block files of a specific file type (for example, .gif), use the following:

User-agent: Googlebot
Disallow: /*.gif$

To prevent pages on your site from being crawled, while still displaying AdSense ads on those pages, disallow all bots other than Mediapartners-Google. This keeps the pages from appearing in search results, but allows the Mediapartners-Google robot to analyze the pages to determine the ads to show. The Mediapartners-Google robot doesn't share pages with the other Google user-agents. For example:

User-agent: *
Disallow: /

User-agent: Mediapartners-Google
Allow: /


For more details see this, and user-agents for Google here.

Vote Up Vote Down