Mobile app version of vmapp.org
Login or Join
Hamaas447

: What's wrong with my robots.txt file to block Google? I have a website and I don't want this website to be visible on Google. So I deployed a robots.txt file. But the homepage of the website

@Hamaas447

Posted in: #RobotsTxt

I have a website and I don't want this website to be visible on Google. So I deployed a robots.txt file. But the homepage of the website is still visible on Google. Here is the full code of my robots.txt copy-pasted below-

User-agent: *
Disallow: /cgi-bin/
Disallow: /private/
Disallow: /tmp/
Disallow: /_/
Disallow: /cgi-bin/
Disallow: /index.html
Disallow: /images
Disallow: /about.html

10.02% popularity Vote Up Vote Down


Login to follow query

More posts by @Hamaas447

2 Comments

Sorted by latest first Latest Oldest Best

 

@Courtney195

Use Disallow: / to block your entire domain from being crawled by google bots.

10% popularity Vote Up Vote Down


 

@Bethany197

If you don't want to allowed to crawl your website on Google search result, then use this robots.txt

User-agent: *
Disallow: /


These will block all directories.

But if many website links to your website, then Google will start displaying this snippet in future.



Now if you don't want to index that website competely, then use noindex meta-tags/HTTP headers. That meta tags simply no index your all pages, but it is allowed to crawl. So if your main concern is about to, not visible in search result, then I highly recommended to use noindex tags.

But don't use it both, because when the site is blocked by robots.txt then Google will not going to see your meta-tags/http headers.

10% popularity Vote Up Vote Down


Back to top | Use Dark Theme