: How to prevent CDN content URLs being indexed by Google Well, robots.txt prevent crawling and meta robots tag in HTML (or) X-Robots-Tag HTTP header prevents indexing (and other functionalities

@Hamm4606531

Posted in: #Cdn #Indexing #RobotsTxt #Seo #WebCrawlers

Well, robots.txt prevent crawling and meta robots tag in HTML (or) X-Robots-Tag HTTP header prevents indexing (and other functionalities available too).

So, even when a URL is disallowed in robots.txt, it can be indexed in Google if it is referenced somewhere using an anchor tag.

So, my question is, how can I prevent public CDN URL being indexed by Google?

For example, I upload an image on Facebook which is private. But, the CDN URL holding the data is actually public, so anyone with the complete link can access the content. How does Facebook prevent these URLs being indexed?

Example textual URL: scontent.fmaa1-2.fna.fbcdn.net/v/t1.0-9/21616414_867163746798211_8462429064810946636_n.jpg?oh=8964e784c64486e307a0fac58e66d79a&oe=5A3C658E
Now, I posted the above URL here. Will google index this URL? Update:Yes, this is indexed as a text content

How about this anchor URL: FBCDNLINK.

Will this be indexed if this is referenced this way?

Update:

When I say "link indexing", I'm not talking about this kind of content search indexing but like this kind of link indexing.

Example: This URL is actually disallowed in the site's robots.txt but you can see that it is actually indexed.

My question is, how does Facebook prevent the FBCDN URL above being indexed while you search like this in Google: site:fbcdn.net inurl:21616414_867163746798211_8462429064810946636_n.jpg

This didn't work, and my question is how does Facebook do this.

Reference 1:

Reference 2: support.google.com/webmasters/answer/6062608?hl=en
Reference 3: tools.seobook.com/robots-txt/ (Check the table at bottom)



TestDingDongBellPleaseIgnoreThis

Vote Up Vote Down

---

Login to post a comment!