: If an visitors IP address contains "google" or a similar keyword, does this mean they were a crawler? I have a huge list of IP addresses recorded from various visitors to a website. A huge

@Angela700

These are more commonly referred to as crawler user-agents. The subdomains of the agents normally refer to a server ID (sometimes including the IP address).

You can find a comprehensive list at www.user-agents.org/

Vote Up Vote Down

@Welton855

That looks genuine, as it's fairly hard to spoof a source domain name -- crawlers can generate a lot of traffic. They often check pages for changes quite frequently.

You can slow down many of them with a non-standard (but fairly well supported) addition to the Robots Exclusion Protocol -- create a file called robots.txt that's served from your web server's root directory with the following contents:

User-agent: *
Crawl-delay: 60


Where the number on the second line is the number of seconds you want each crawler to wait between page loads on your site. (If you've already got a robots.txt file you'll need to modify it instead.)

Vote Up Vote Down