3 ms·
I've been aggregating my news for my own use recently so I can keep a database of articles and grab articles from more sources than just RSS (some are parsed ou
by wybiral 9y ago
I've been aggregating my news for my own use recently so I can keep a database of articles and grab articles from more sources than just RSS (some are parsed out of HTML).
The Python code for some of the sources is here: https://github.com/wybiral/stream-sources https://github.com/wybiral/stream-sources
On my network there's a bunch of those stream sources feeding into a Golang server that stores the articles and aggregates the ones I'm interested in for real-time updates/alerts.
The nice thing is that I can attach temporary scrapers with alerts to monitor status pages and other updates that don't use RSS or anything.
IMO WebSocket API's or streaming HTTP make more sense than RSS these days. But RSS can easily be converted into a format like that if you store the recent history and only push new content.
- __bee 9y agoGenerally speaking, is it legal storing RSS content (if the RSS contains the article)
- wybiral 9y agoIt's for my own use. Also I don't see why it would be any different than leaving it open in my browser or something. The articles here are mostly title, link, summary too. So not too unlike what someone would share on social media or something.