: What are proper process to disallow a site from google crawl I have done google and get the two solution for disallow whole a site from google crawl. 1: User-agent: * Disallow:

@Alves908

To prevent your whole site from being crawled, then No. 2:

User-agent: *
Disallow: /


This blocks every URL from being crawled. The URL-path following the Disallow: directive is a prefix. If the requested URL starts with this URL-path, it will be blocked. The minimum URL path you can have is / (your home page / document root) - you can't have an empty path (as suggested in comments). When you request example.com, the browser actually requests example.com/ to make the request valid. See my other answer for more information on the trailing slash.

Disallow: by itself (without a path) actually allows everything - the complete opposite!

Reference: www.robotstxt.org/robotstxt.html

Vote Up Vote Down