: You'll need a dedicated server for this - you shouldn't have any problems with most DS hosts. A few are listed here but there are plenty out there so just shop around. The plus side of

@Cofer257

You'll need a dedicated server for this - you shouldn't have any problems with most DS hosts. A few are listed here but there are plenty out there so just shop around.

The plus side of having your own server is you can host your other web sites on there, assuming your crawler is not a resource hog. A URL retrieval every 2-5 seconds would not slow down the server for the other sites.

Alternatively, why not run it from your own computer? Web pages are usually only a few dozen kB, or smaller with gzip, so it probably wouldn't use masses of your bandwidth.

Vote Up Vote Down

Login to post a comment!