: Googlebot fetching lots of URLs with a "workaround" parameter A certain page on our website keeps generating entries like the following on our IIS6 logs: 2011-08-16 00:53:34 W3SVC943830190 10.20.101.146
A certain page on our website keeps generating entries like the following on our IIS6 logs:
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707368055555 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707277777777 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707347222222 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707291666666 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707263888888 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707326388888 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707437500000 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707451388888 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707340277777 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707270833333 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:35 W3SVC943830190 10.20.101.146 GET /abc workaround=407707381944444 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:35 W3SVC943830190 10.20.101.146 GET /abc workaround=407707284722222 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:35 W3SVC943830190 10.20.101.146 GET /abc workaround=407707201388888 80 - 66.249.85.65 - 200 0 0
Doing a reverse DNS lookup, these IP addresses making these calls seems to be a Googlebot. What does "workaround=" on this line mean?
We always seem to be getting "workaround=" calls on 1 particular page only, sometimes multiple times within a few seconds like this example. What is the cause of this and how can I prevent it?
This specific page does make AJAX calls via jQuery to a 3rd party service and they are reporting a spike in calls being made and I am suspecting this bot calls to be the issue.
More posts by @Caterina187
1 Comments
Sorted by latest first Latest Oldest Best
I'm not sure exactly what "workaround" means. It seems likely that the third party AJAX is including that string in data that is returned. Google may be finding it in either HTML or JavaScript content and then deciding that it looks enough like a URL that it should crawl it.
To prevent Googlebot from crawling those URLs on your site, you could add this to your robots.txt file:
User-Agent: Googlebot
Disallow: /*workaround=
It sounds like the AJAX shouldn't be crawled either. In that case, the third party site might have to add something to their robots.txt. You could also put the JavaScript code that calls the AJAX into robots.txt so that Googlebot can't see it:
User-Agent: Googlebot
Disallow: /js/call-third-party-ajax.js
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.