: Do web crawlers/spiders index azure web sites? For somebody who wants their web site to be as discoverable as possible (and who doesn't?), are Microsoft's Azure web sites (azurewebsites.net) a

@Candy875

So, are your websites exactly the same? same content, same html? and the only difference is domain and hosting?

Spiders will crawl any website as long as it is public, and no rules inside robots.txt or metatags explicitly tell them to keep out of the site.

Your problem is due to duplicate content, to Googlebot's eyes, one of your sites is copying content from another site and it doesn't like that, it will give credit to the site it thinks first published that content but will take into account authority, ranking, longevity and it will disregard or it will even penalize the site that's duplicating content.

From Google Webmaster Tools help:


Google tries hard to index and show pages with distinct information.
This filtering means, for instance, that if your site has a "regular"
and "printer" version of each article, and neither of these is blocked
with a noindex meta tag, we'll choose one of them to list. In the rare
cases in which Google perceives that duplicate content may be shown
with intent to manipulate our rankings and deceive our users, we'll
also make appropriate adjustments in the indexing and ranking of the
sites involved. As a result, the ranking of the site may suffer, or
the site might be removed entirely from the Google index, in which
case it will no longer appear in search results.

Vote Up Vote Down