: How do I prevent duplicate indexing by Googlebot-Mobile? I have only one version of my web-site, and it is optimised for both the desktop and the smartphone. I've noticed that Google crawls
I have only one version of my web-site, and it is optimised for both the desktop and the smartphone.
I've noticed that Google crawls each page exactly twice: once with Googlebot/2.1, and again with Googlebot-Mobile/2.1:
% fgrep "/www/drupal6/filefield " /var/www/logs/ports.su/ports.su.access.log
66.249.76.223 - - [12/Mar/2013:07:46:42 -0700] "GET /www/drupal6/filefield HTTP/1.1" 200 2365 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.223 - - [12/Mar/2013:13:59:23 -0700] "GET /www/drupal6/filefield HTTP/1.1" 200 2365 "-" "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"
I don't think blocking either of these bots in /robots.txt would be appropriate, as it might remove my site from their respective index.
However, don't they share their indices, too?
How do I tell Google that I only have one site?
Looking for something that's the opposite of Vary: User-Agent.
More posts by @Murray432
2 Comments
Sorted by latest first Latest Oldest Best
You are already telling them because you have one site using a single set of URLs.
They visit twice because each bot evaluates differently. In the end it goes in the same index but the metadata is different which lets them serve different results to standard and mobile queries.
Having Googlebot/2.1 and Googlebot-Mobile/2.1 is a good thing, you should not block either.
It's important to note that when viewing Google search results on a mobile phone they tend to vary from a Desktop Google search since Google favors sites that are viewable on a mobile phone. Blocking Googlebot-Mobile/2.1 may prevent people finding your site when they use Google search on their phone.
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.