: Content loaded via XMLHttpRequest (xhr): How to make google aware of it? According to google itself and another question here, google's bots execute JavaScript on crawled sites and index the rendered

@Pope3001725

Posted in: #Ajax #Seo

According to google itself and another question here, google's bots execute JavaScript on crawled sites and index the rendered content.

However,


[a]ccording to [the] experience [of at least this one company], while most of the JavaScript-based functionality is now understood by Googlebot, it consistently fails to crawl the content retrieved by XMLHttpRequest API from an external source - or anything built on top of, or related to this API. This behavior is present in pure JavaScript, jQuery, AngularJS, or other modern JavaScript frameworks. Whenever you need to pull content from an "external" URL or call a REST API endpoint to fetch some data, chances are that it will not be crawled and indexed properly.


(emphasis mine) This behaviour we are now facing as well:


We have a react app that integrates into a client's website, whose landing page is a list of things with links to a subpage for each (<a data-reactid="0.1.2.3.4" href="/some/site/itemid/9876">)
The landing page is rendered correctly in the Google search console ("This is how Googlebot saw the page" visually equals "This is how a visitor to your website would have seen the page")
Content of the landing page can be found in google's search results.
No content of any subpage can be found via google in the near term (weeks later, content of a fraction of the subpages is).


Here is a demo react app with react/js-generated links and subpage that were correctly indexed. Key difference is, as mentioned, that we retrieve the content of the subpages from an external server. There is no way around that - we have the content, but the website that shall render it only includes our js script and an element to render it into, we have no further control over the website, much less the server it runs on.

Is the above quote correct or am I missing something?

Vote Up Vote Down

Login to post a comment!