I spent the last two years building, deploying, and maintaining a pretty large custom search engine based on scraping. I agree with most/all of the business com

I spent the last two years building, deploying, and maintaining a pretty large custom search engine based on scraping. I agree with most/all of the business comments made in the thread. From a technical perspective the main thing to keep in mind is that scraping is a dirty process, more so when you're scraping from smaller firms that often have out of date and quite horrible sites. It's not something you can build and just run. Sites will break, fail to respond, change their markup, etc. You system has to be very tolerant, or you'll be in babysitting mode 24 x 7.