: From John Conde's reply to a question of mine, see IP Lists. That site lists known IPs used by each search engine. Filtering by user-agent is not reliable because it can be faked extremely

From John Conde's reply to a question of mine, see IP Lists. That site lists known IPs used by each search engine.

Filtering by user-agent is not reliable because it can be faked extremely easily. Google suggests looking at the user-agent and then doing a DNS lookup. This way, if they add new IPs, the filtering does not break. You obviously have to update your DNS valiation code when new search engines or bots appear by that would not be too often.

10% popularity Vote Up Vote Down

: How To Track Down and Stop Rogue Bots? Most of the bandwidth of one site is being consumed by an unidentified bot. According to AWSTATS it says: Unknown robot (identified by 'bot*') consumed

@Welton855

Posted in: #Bandwidth #WebCrawlers

2 Comments

: Why is a site I've never heard of (and don't own) showing up in my Adsense reports? Last week a registered a domain (call it my-domain.com) and put some Adsense ads on it. Today I noticed

@Welton855

Posted in: #Google #GoogleAdsense

4 Comments

: If you are currently using www.example.com for yourself and giving users username.example.com, then additionally allowing users to create username.com as an alias of username.example.com doesn't

@Welton855

0 Comments

: Most immediate CDN I'm interested in the user experience effects of a near-instantly loading home page. Are there CDNs specifically designed to serve tiny data, like a 500-byte HTML page, to

: From John Conde's reply to a question of mine, see IP Lists. That site lists known IPs used by each search engine. Filtering by user-agent is not reliable because it can be faked extremely

More posts by @Welton855

: How To Track Down and Stop Rogue Bots? Most of the bandwidth of one site is being consumed by an unidentified bot. According to AWSTATS it says: Unknown robot (identified by 'bot*') consumed

: Why is a site I've never heard of (and don't own) showing up in my Adsense reports? Last week a registered a domain (call it my-domain.com) and put some Adsense ads on it. Today I noticed

: If you are currently using www.example.com for yourself and giving users username.example.com, then additionally allowing users to create username.com as an alias of username.example.com doesn't

: Most immediate CDN I'm interested in the user experience effects of a near-instantly loading home page. Are there CDNs specifically designed to serve tiny data, like a 500-byte HTML page, to

Login to post a comment!

0 Comments

Back to top | Use Dark Theme