3 ms·

You might want to take a look at Samuel Clay's NewsBlur project: https://github.com/samuelclay/NewsBlur https://github.com/samuelclay/NewsBlur and see how he ha

• conesus 16y ago
  As others have said, task queues (look into Celery, which is in Python), Mark Pilgrim's feedparser, and make sure you don't fetch more often than you need to. A few thousand feeds is fine, but if you grow in the hundreds of thousands of feeds, if you want to update them more than once a day, you're going to have to pass in the cache controls (etags and last-modified dates). Even then, 50,000+ feeds is straining the limits of a single DB. If you were to reach that point, then I recommend moving to a nosql db (like mongo, which is what I use for NewsBlur), and sharding it so you can read/write to more feeds without a problem. All of your analysis will have to be in a MapReduce, so it can be sent to different shards, but that's not too difficult to learn how to do. NewsBlur has a few examples of how to do this. I used Python, but Ruby would also be a fine choice. Although I'm not sure what Ruby libraries you would use.