Mobile app version of vmapp.org
Login or Join
Angela700

: How to identify if IP address is really google's IP From what I learned in my server logs, IP addresses starting with 64.249 belong to google. When I do a host lookup on my computer for

@Angela700

Posted in: #Google #IpAddress #WebCrawlers

From what I learned in my server logs, IP addresses starting with 64.249 belong to google. When I do a host lookup on my computer for google I receive 11 different IP addresses all starting with 74.125.226.

I then go here www.iplists.com/ and find out there's a bunch more IP addresses for google.

I even did a whois lookup in a command prompt for google and see "Aborting search 50 records found ....." along with a whole bunch of servers with unusual names such as GOOGLE.COM.ANGRYPIRATES.COM and GOOGLE.COM.AFRICANBATS.ORG. along with different IP addresses I haven't seen before such as 91.218.229.20.

What I want to do is whitelist real google IP addresses. Should I just go ahead and white list every single IP address I found thus far or are there other precautions I should take before white listing a google address?

I don't want to rely on others searching for IP addresses for me such as the source www.iplists.com/ because the IP addresses can change at any time and I want to update my white list frequently.

10.02% popularity Vote Up Vote Down


Login to follow query

More posts by @Angela700

2 Comments

Sorted by latest first Latest Oldest Best

 

@Jamie184

From the official docs to verify Googlebot / Google:

Note that Google does not recommend using a static "whitelist".


You can verify if a web crawler accessing your server really is Googlebot (or another Google user-agent). This is useful if you're concerned that spammers or other troublemakers are accessing your site while claiming to be Googlebot. Google doesn't post a public list of IP addresses for webmasters to whitelist. This is because these IP address ranges can change, causing problems for any webmasters who have hard-coded them, so you must run a DNS lookup as described next.

To verify Googlebot as the caller:


Run a reverse DNS lookup on the accessing IP address from your logs, using the host command.
Verify that the domain name is in either googlebot.com or google.com.
Run a forward DNS lookup on the domain name retrieved in step 1 using the host command on the retrieved domain name. Verify that it is the same as the original accessing IP address from your logs.



(Emphasis my own)

See the Google docs for a worked example: support.google.com/webmasters/answer/80553?hl=en

10% popularity Vote Up Vote Down


 

@Sherry384

First things first.

There are two domain names that we are talking about; Google.com and Googlebot.com. Doing a reverse look-up for Google.com would be misleading.

Here is an answer I gave quite a while ago where I did an audit of the Google IP addresses used to crawl my site over 8 years. This is not the be-all end-all list, but could give an historical view.

What does Google use 173.194.90.xxx IP's for? Why is it not a googlebot.com pointer?

In this list, you can see the IP address range 64.249 you are seeing is correct.

I did a quick audit for this answer. This is my current white-list for googlebot (agent name). Please forgive the wildcarding. I have updated my code to understand ranges more efficiently, but have not updated the list. Sorry. In this case, I just use a simple regular expression to compare the IP address to the list with a minimum of trickery to replace the * with something that regex understands. I can compare an IP address to the entire list using my code so it is still fast enough.

64.233.160.*
64.233.161.*
64.233.162.*
64.233.163.*
64.233.164.*
64.233.165.*
64.233.166.*
64.233.167.*
64.233.168.*
64.233.169.*
64.233.170.*
64.233.171.*
64.233.172.*
64.233.173.*
64.233.174.*
64.233.175.*
64.233.176.*
64.233.177.*
64.233.178.*
64.233.179.*
64.233.180.*
64.233.181.*
64.233.182.*
64.233.183.*
64.233.184.*
64.233.185.*
64.233.186.*
64.233.187.*
64.233.188.*
64.233.189.*
64.233.190.*
64.233.191.*
66.249.64.*
66.249.65.*
66.249.66.*
66.249.67.*
66.249.68.*
66.249.69.*
66.249.70.*
66.249.71.*
66.249.72.*
66.249.73.*
66.249.74.*
66.249.75.*
66.249.76.*
66.249.77.*
66.249.78.*
66.249.79.*
66.249.80.*
66.249.82.*
66.249.83.*
66.249.84.*
66.249.85.*
66.249.86.*
66.249.87.*
66.249.88.*
66.249.89.*
66.249.90.*
66.249.91.*
66.249.92.*
66.249.93.*
66.249.94.*
66.249.95.*
74.125.*
173.194.*


GOOGLE.COM.ANGRYPIRATES.COM and GOOGLE.COM.AFRICANBATS.ORG are what is known as a domain or sub-domain hack. These do not belong to Google at all. This is something anyone can do to any domain name at any time. Without getting into the details of how this is done, you can find information on the net. I do not want to encourage this behavior by explaining it here.

10% popularity Vote Up Vote Down


Back to top | Use Dark Theme