: Specifying crawlers to not crawl Links which are dependent on external API's I have links on my site which lead to internal pages that depend on first bringing data from an external API. This

@Cofer257

Adding that meta tag will prevent search engines from crawling your entire site, so you should avoid that! Adding pages to the sitemap should allow them to be crawled and indexed (since you are not telling search engines to ignore the actual pages). But if you have no links to them on your own site (as search engines see it) then they will not rank well, or at all.

One solution would be to use robots.txt and block the URLs that use the API, assuming they follow a standard format or it is easy to generate the list of things to block.

However, the better solution would be to spider the site yourself and make sure all pages you link to are already generated and in your database. This way the pages will be fast for users (your main concern of course) and search engines as well.

Vote Up Vote Down