: Anyone use Spiders apart from search engines and spammers? Are spiders ever used by anyone other than search engines and spammers? If someone wanted to collect information from the web, would

@Angie530

There are many, many potential uses for web crawlers - a review of Wikipedia's list of Open Source Web Crawlers or examples of Spidering Hacks provided by O'Reilly may help illustrate the numerous ways webmasters and programmers utilize others' services and information.

In addition to "friendly" spiders, you can also expect to see plenty of spambot information harvesters, scrapers, contact form spammers, and vulnerability scanners (like Nessus) in your site's access logs.

Anyone who is familiar with scripting and has an interest in other sites' services can implement a spider of some variety.

Vote Up Vote Down

@Welton855

You could use a spider to crawl your own site looking for broken links.

Vote Up Vote Down

@Pope3001725

Some reasons a bot may be used that come to mind for me are:


Some directories will crawl a website for its meta information (usually after a site has been submitted to the directory)
Contextual ad services like Adsense use bots to figure out what a page is about so relevant ads can be displayed
Webmasters can use bots to find potential linking partners. This may fall into the spammer category but I want to list it for the sake of people who don't use it maliciously.
Webmasters can use bots to verify that a link posted in a link exchange has been and still is on the partner's site

Vote Up Vote Down