: How can I block visitors located in the U.S. while still allowing the Googlebot? Due to regulation and legal issues we cannot allow U.S. based visitors to access our website. For this reason
Due to regulation and legal issues we cannot allow U.S. based visitors to access our website. For this reason we redirect all our U.S. traffic to a specialized page.
While we do have these restrictions, our (financial) products are perfectly legal and regulated in other countries in Europe, Middle East and Asia.
Since we don't want to be considered as cloaking, we redirect all IP's - humans and bots alike.
As far as I understand most, if not all, Googlebot's IP ranges come from the U.S. So my question is how can we allow Google to index our website for the benefit of non-U.S. based users and avoid cloaking (or post-cloaking). Can Googlebot access from non-U.S. IP ranges? What would the preferred course of action be?
More posts by @Nimeshi995
2 Comments
Sorted by latest first Latest Oldest Best
You will probably have problems with Google indexing if you try and block US traffic. Googlebot crawls mainly from the US.
Google says
"Google does not, right now, have any crawling that happens from
non-US IP addresses. Right now - we only crawl from the United States"
Source
A javascript solution has been suggested by google a few years ago stating :
One potential solution would be to use a JavaScript-based interstitial
that verifies the IP address and otherwise blocks access to your site.
I assume you have to use JavaScript within your site, correct? If so,
there would be no simple way for a user to selectively block the
JavaScript interstitial and allow the JavaScript casino content.
Assuming the JavaScript is in an external file that is disalllowed
through your robots.txt file, Googlebot would not be able to view the
interstitial and would be able to crawl the site normally.
Source productforums.google.com/d/msg/webmasters/thqhs2Mrmtw/LdSu7vj8Oi4J
Googlebot does have IP addresses located in other countries. For a (likely incomplete) list, see the following link and click on each IP address listed there to see its country of origin:
Bots vs Browsers - User Agent Details for "+http://www.googlebot.com/bot.html"
You could add these IP's, and others you might find elsewhere, as exclusions to your IP blocking rules. However, there is no guarantee that those specific IP addresses will crawl your site if you block the U.S. based ones.
Therefore to help with this, you should indicate a Geotarget for a country outside of the U.S. in Google Webmaster Tools, as detailed here: Geotargeting
Using a country code top-level domain (ccTLD) might also help get your site listed in country specific versions of Google search, such as google.de (for Germany), and hopefully crawled by servers located outside the U.S.
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.