: Database of websites for a search engine I'm planning to make a search engine, so I've downloaded this script. It works well, but I have to add a list of websites manually in a php file

@Courtney195

This may help : Where can I download list of all .com domains registered in the world

Also as of last year there were only 70 million sites which Yahoo indexed.

So you get all the domains and then fetch their Alexa rankings and start indexing in ascending order of rankings.

Vote Up Vote Down

@Radia820

Probably the fastest way to pull together such a list is to scrape DMOZ and spider from there on out.

Vote Up Vote Down

@Jessie594

There are over 600M websites in the world, and that number is probably going up 1000s a day. I don't mean to be an ass, but a script and any webserver you come up with isn't going to be able to be a remotely fast or effective search engine.

Use something like Google Custom Search to setup a a Google frontend just for your websites, or if you actually wanted to rival Google just give up now.

Vote Up Vote Down