: Google is indexing unintended content that is either unpublished or a secret part of site Google is indexing parts of my site that I haven't linked to, it's a mobile version of website that

@Smith883

If you have installed google analytics on your site, then this is almost definitely how it knows about them. I don't know the code off the top of my head but if I get a chance later I'll search it out and edit this post but there is code you can put on your pages to prevent crawling on pages you want to be private.

Vote Up Vote Down

@Ogunnowo487

It is in Google's interests to index all of the public web, including content that contains no hyperlinks from existing indexed content.

The use of m.example.com or example.com/m/ for the URL of a mobile version of a site is common.

It is therefore reasonable to assume that Google's crawler will see if m.example.com or example.com/m/ exists and attempt to index such content if found.

It is also reasonable to assume that Google Mobile will attempt to determine if a suitable-looking m.example.com or example.com/m/ URL exists for known content and present this as a link to a user in search results. There is no reason to think such divination would not feed back to the crawler.

Vote Up Vote Down

@Gretchen104

There are various theories as to how Google knows what to crawl. It could be that someone linked to your mobile version. It could be that Google tried random urls and came across the /m version of your site. I'm not aware that they say they won't use URLs from their analytics data.

Yes they do follow those rules: googlewebmastercentral.blogspot.com/2007/03/using-robots-meta-tag.html.

Vote Up Vote Down