: Why is GoogleBot crawling HTML pages as JSON using as referrer the one from my android app? Recently, I started finding in my logs errors caused by missing templates for pages that reasonably

@Reiling115

Posted in: #Android #Googlebot #WebCrawlers

Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e.


REQUEST_PATH : /foo/bar.json
HTTP_HOST : myapp.com HTTP_FROM : googlebot(at)googlebot.com
HTTP_REFERER : android.app.1.8.2 (the one we use in our android app)
HTTP_USER_AGENT : Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

myapp.com/foo/bar is a valid page but there is nothing being served as json.

As far as I can tell, the invalid link with the .json extension does not appear anywhere (and in fact, I cannot find the string indexed in google at all).

What could be causing this?

Is it possible google is looking at json data in our android app (where the link without json might legitimately appear) and crawling it?

Is there any way I could avoid this, to avoid my logs being spammed with spurious requests?

Is it even likely that this is in fact google rather than something else?The request seems like a legitimate googlebot request based on the IP address.

Vote Up Vote Down

Login to post a comment!