: Prevent azure subdomain indexation Let me explain my situation, I have an azure website (with azurewebsites.net sub domain), and a custom domain.com, built with asp.net MVC Both are being indexed

@Chiappetta492

If you are using IIS, you can add IIS rewrite rules to your web.config to specify which type of robots.txt to return, depending on the subdomain the user (and thus the crawler) is browsing to. You can specify special HTTP_HOST pattern conditions to specify which robots.txt file should be used for which domain.

An article which explains this perfectly: www.tekcent.com/articles/2013/dynamic-robots-exclusion-file-in-aspnet/

Vote Up Vote Down

@Hamaas447

If you don't want to separate robots.txt or you don't want to add metatags noindex then your options are very limited; one could be to password protect your azure site, but that will lead to restricted visitors access.

Some about this here.

On the other hand, is your robots.txt too complex? does it have tons of rules? why do you give it too much importance? on regular basis it is just something like this:

User-agent: *
Disallow:


Just delete it from your local folder, and never upload it via ftp or whatever you are using, create an independent robots.txt file for each site and let each one grow apart.
Put this simple robots.txt into your azure site:

User-agent: *
Disallow: /


And put your "complex" robots.txt on the robots allowed site:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/

Vote Up Vote Down