: If you want to block search engines from crawling and/or indexing your pages you can use a few methods: 1) Use robots.txt 2) Use meta tags <meta name="robots" content="noindex, nofollow">

@Sarah324

If you want to block search engines from crawling and/or indexing your pages you can use a few methods:

1) Use robots.txt

2) Use meta tags

<meta name="robots" content="noindex, nofollow">


3) Use http headers

Header set x-robots-tag: noindex


4) Use rel="nofollow"

<a href="http://www.example.com/sample.html" rel="nofollow">Link to page I don't want indexed</a>


5) Place the content behind a login. Search engines (generally) do not submit forms nor create accounts at websites.

6) Block the bots of all of the major search engines using .htaccess but that would be tedious and prone to error if they change whatever it is you would use to identify them (e.g. ip address, user agent).

Vote Up Vote Down

Login to post a comment!