Mobile app version of vmapp.org
Login or Join
Caterina187

: Googlebot fetching lots of URLs with a "workaround" parameter A certain page on our website keeps generating entries like the following on our IIS6 logs: 2011-08-16 00:53:34 W3SVC943830190 10.20.101.146

@Caterina187

Posted in: #Googlebot #Iis6

A certain page on our website keeps generating entries like the following on our IIS6 logs:

2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707368055555 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707277777777 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707347222222 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707291666666 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707263888888 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707326388888 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707437500000 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707451388888 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707340277777 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707270833333 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:35 W3SVC943830190 10.20.101.146 GET /abc workaround=407707381944444 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:35 W3SVC943830190 10.20.101.146 GET /abc workaround=407707284722222 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:35 W3SVC943830190 10.20.101.146 GET /abc workaround=407707201388888 80 - 66.249.85.65 - 200 0 0


Doing a reverse DNS lookup, these IP addresses making these calls seems to be a Googlebot. What does "workaround=" on this line mean?

We always seem to be getting "workaround=" calls on 1 particular page only, sometimes multiple times within a few seconds like this example. What is the cause of this and how can I prevent it?

This specific page does make AJAX calls via jQuery to a 3rd party service and they are reporting a spike in calls being made and I am suspecting this bot calls to be the issue.

10.01% popularity Vote Up Vote Down


Login to follow query

More posts by @Caterina187

1 Comments

Sorted by latest first Latest Oldest Best

 

@Eichhorn148

I'm not sure exactly what "workaround" means. It seems likely that the third party AJAX is including that string in data that is returned. Google may be finding it in either HTML or JavaScript content and then deciding that it looks enough like a URL that it should crawl it.

To prevent Googlebot from crawling those URLs on your site, you could add this to your robots.txt file:

User-Agent: Googlebot
Disallow: /*workaround=


It sounds like the AJAX shouldn't be crawled either. In that case, the third party site might have to add something to their robots.txt. You could also put the JavaScript code that calls the AJAX into robots.txt so that Googlebot can't see it:

User-Agent: Googlebot
Disallow: /js/call-third-party-ajax.js

10% popularity Vote Up Vote Down


Back to top | Use Dark Theme