: Can you create a private website without password protection that would not be indexed? I'd like to create a "private" website that clients can visit by typing in the provided direct URL (on

@Nimeshi995

Probably not.

If any of your clients post the URL in just about anything, search engines will likely discover it. They may also find it while crawling domain registration sites and the like...

As Google states here:


If you need to keep confidential content on your server, save it in a
password-protected directory. Googlebot and other spiders won't be
able to access the content. This is the simplest and most effective
way to prevent Googlebot and other spiders from crawling and indexing
content on your site.


Even if you try to block crawling using robots.txt, noindex meta tags, or X-Robots-Tag HTTP headers, any mention of your site anywhere may result in crawling and indexing it by various search engines. As Google indicates:


It's important to note that even if you use a robots.txt file to block
spiders from crawling content on your site, Google could discover it
in other ways and add it to our index. For example, other sites may
still link to it. As a result, the URL of the page and, potentially,
other publicly available information such as anchor text in links to
the site, or the title from the Open Directory Project, can appear in
Google search results. In addition, while all respectable robots will
respect the directives in a robots.txt file, some may interpret them
differently. However, a robots.txt is not enforceable, and some
spammers and other troublemakers may ignore it. For this reason,
we recommend password-protecting confidential information (see above).

Vote Up Vote Down

@Kevin317

The easiest way would be to use htaccess to send an HTTP header to tell the search engines not to index any of those pages. Put the following code in an htaccess file in your root web directory:

Header: "X-robots-tag: noindex"


This will apply this header to every page in that directory and subdirectory below it and prevent those pages from being indexed.

You can read more about this in Google's Webmaster Blog.

Vote Up Vote Down