3 ms·
We built a backend similar to this for our NewsRoom mobile client. (Android and Pre) Actually used some genetic algorithms to do the training for our content e
by nicpottier 16y ago
We built a backend similar to this for our NewsRoom mobile client. (Android and Pre) Actually used some genetic algorithms to do the training for our content extraction, one of the more fun projects I've done.
Word of warning, if it takes off, you basically start turning into someone who is both caching and harvesting the web every 15 minutes. There is an incredibly long tail on RSS feeds and it starts killing you to keep them all up to date. Storing and serving it is no big deal, but harvesting actually turns into real money when you figure out total bandwidth used. (we harvest about ~30,000 feeds every 15 minutes)
- lurchpop 16y agohow do you handle scale like that? Do you have a hadoop cluster or something? How many concurrent do you download?
- nicpottier 16y agoAmazon EC2, MongoDB, S3. The EC2 instances scale with how many stale feeds we have, but it is usually less than 2. Just checked and we have ~25k feeds in the system, though not all are deep harvesting as we call it. Note we do a few things over just extracting the full content as well, we also try to grab out images and create a pleasing thumbnail using face detection etc.. So that probably slows things down a good deal as well.
- tianyicui 16y agoI really wish one day Google will sell a page harvesting service. It can certainly profit since the cost is neglectable. But how big the market is?