: How should I protect "secret" links I send in Emails from being indexed by search engines? I have noticed that bing/msnbot tries to index pages that were only ever linked to in one single

I have noticed that bing/msnbot tries to index pages that were only ever linked to in one single email.

My guess is, that some kind of browser toolbar is responsible for this.

Apart from robots.txt, is there something I can or should do to prevent search engines from indexing private pages, i.e. pages that could only ever be discovered by sniffing "private" content like emails or password protected pages my users browse to?

10.01% popularity Vote Up Vote Down

: Make 404 error not redirect when on different domain/subdomain I have a PHP file on my error subdomain (https://errors.example.com). And this is meant to serve as the error page for all subdomains.

@Jennifer507

Posted in: #Htaccess #Php

1 Comments

: Is it ok to block search engine bots from China and Russia on USA based eCommerce site? After blocking all traffic from China and Russia ( using Incapsula ) on an eCommerce application which

@Jennifer507

Posted in: #Performance #Seo #WebTraffic

1 Comments

: Cloudflare DNS: How to 301 Redirect all traffic from Sub Domain to Main Domain with URL path? I am using cloudflare to manage my domain, and I wanted to add a subdomain. I tried to do this:

@Jennifer507

Posted in: #Cloudflare #Dns #Domains #Subdomain

1 Comments

: CloudFlare DNS: How to setup a GitHub Repository to a custom sub domain? I have a github pages site, and I want to add it as a subdomain to my cloudflare domain without using a page rule

@Jennifer507

Posted in: #Cloudflare #Cname #Dns #Github

1 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Jamie184

If you specifically want to prevent a page/URL from being indexed then include a noindex robots meta tag on HTML responses (or equivalent X-Robots-Tag HTTP response header).

Do not block the page with robots.txt. The robots.txt file primarily prevents the page from being crawled (which has the added side-effect that it probably doesn't get indexed). However, pages blocked by robots.txt can still be indexed if they are linked to (a link-only result appears in the SERPs). And blocking crawling with robots.txt prevents the search engine bots from seeing the robots noindex meta tag. Including the URL in robots.txt can also be an advert to "bad" bots.

However, if the page contains sensitive information or must only be seen by the intended recipient of the email then it should be blocked by some kind of authentication.

10% popularity Vote Up Vote Down

Feed

: How should I protect "secret" links I send in Emails from being indexed by search engines? I have noticed that bing/msnbot tries to index pages that were only ever linked to in one single

More posts by @Jennifer507

: Make 404 error not redirect when on different domain/subdomain I have a PHP file on my error subdomain (https://errors.example.com). And this is meant to serve as the error page for all subdomains.

: Is it ok to block search engine bots from China and Russia on USA based eCommerce site? After blocking all traffic from China and Russia ( using Incapsula ) on an eCommerce application which

: Cloudflare DNS: How to 301 Redirect all traffic from Sub Domain to Main Domain with URL path? I am using cloudflare to manage my domain, and I wanted to add a subdomain. I tried to do this:

: CloudFlare DNS: How to setup a GitHub Repository to a custom sub domain? I have a github pages site, and I want to add it as a subdomain to my cloudflare domain without using a page rule

Login to post a comment!

1 Comments

Back to top | Use Dark Theme