5 ms·

Combine multiple RSS feeds into a single feed, as a service

61 points by 8organicbits 2y ago

g15jv2dp 2y ago
I cannot see any use I'd have for that. All the RSS readers I've used in my life have supported reading from multiple feeds. Could someone enlighten me?
- bertman 2y ago
  I guess this moves complexity away from the client and enables a dead simple setup (maybe for non-technical friends and family?) with just a single big feed, like they already know from TikTok and the likes.
- batch12 2y ago
  Maybe this could be used to offer aggregated feeds as a service. It could be further refined to filter content based on criteria either provided by the user or predefined by the service. An example could be something like an infosec feed with curated sources and maybe an optional refinement for content related to vulnerabilities. This could also allow users to develop/fork the aggregated feeds and share them with a community.
  - rambambram 2y ago
    This in combination with the <source> element from an item is a killer combo.
- PeterStuer 2y ago
  If you use rss as a middleware protocol, it saves you in client configuration. You deal with the updates and changes on the sources, for your client app it is just one never changing feed. Tbh, I run specialized vertical services aggregating 100's of sources (rss as well as other), and the combination towards a single feed is trivial.
- crazygringo 2y ago
  I am equally baffled. It's a pet peeve of mine seeing software projects that describe what they do, but without providing even a single use case. Or even what motivated the developer. Why does this thing exist? There is probably a reason and then I could learn something.
  - sussmannbaka 2y ago
    A bunch of friends and I once toyed around with the idea of replacing microblogging through ActivityPub with one RSS feed per user. To then display them in one combined feed, I just added all of them to a category in my RSS reader, the way I already did with topics before. One of my friends' RSS reader didn’t have that option (grouping by category yes but no combined feed) so they instead combined the feeds into a single RSS feed and added that to their RSS reader. In the end it was kinda finicky so we went back to ActivityPub.
    - jayemar 2y ago
      I'm not sure if this is the functionality you are looking for, but Mastodon has built in RSS feeds per user. You just to add .rss to the user link. So, for example, if you had an account on mastodon.social, thered be an RSS feed of your posts at mastodon.social/@sussmannbaka.rss
      - trekz 2y ago
        Yeah, there are also third-party services that provide RSS feeds for additional Mastodon data like tags as well. e.g. https://openrss.org/mastodon.social/tags/assange https://openrss.org/mastodon.social/tags/assange
- harryvederci 2y ago
  I think koreader[0] has an RSS feature where you can fill in some RSS feeds and it converts feed items to epubs. It's nice, but I'm using a different reader on my laptop, so then syncing (of which feeds you follow) becomes a thing to do. Instead of following a bunch of them and having to sync them I guess you could just follow the combined one and be done with it. If you have multiple feed readers, each with their own way to input feeds, I think something like rsscombine might be an easy solution. [0] https://koreader.rocks/ https://koreader.rocks/
- 8organicbits 2y ago
  Check out planets[1] as one use case. Projects like Debian, Mozilla, Python, and many more aggregate feeds into a planet to promote the blogs of members or other people who work in their space. Being able to subscribe to "Debian blogs" in one step is helpful versus discovering each blogger individually. There's a big list here [2], I think two use the linked code. [1] https://indieweb.org/planet https://indieweb.org/planet [2] https://github.com/ArchiveTeam/urls-sources/blob/master/43200_blog_aggregators.txt https://github.com/ArchiveTeam/urls-sources/blob/master/4320...
- insom 2y ago
  I saw this on HN and though “great I was just about to write this but now I don’t need to” - I have a legacy blog, a link blog and another thing I update which all produce RSS and I want to create one big feed of stuff I write, regardless of which platform I chose to put it on. Seems like I can use this for that.
  - brokenmachine 2y ago
    But almost every RSS reader has this feature. I use FreshRSS and you just put all the feeds in one category and then you can click on that category to show all the feeds together.
    - saulpw 2y ago
      The reader has to add every RSS feed. What if I as the producer of those feeds want to ship a combined feed so they don't have to do that extra work? Or what if I'm a curator of RSS feeds and want to release one combined "low-tech computer news" feed sourced from 100 smaller low-tech computer news feeds?
      - rakoo 2y ago
        This is already solved with dynamic opml, which freshrss implements
- richardw 2y ago
  I’ve wondered if this existed, to simplify writing a system that parses and processes RSS. Frankly I’d go a bit further and spit out events, so people can code “give me the next entry” without having to mess with polling lots of rss endpoints. Just tell me when there’s a new entry. Or turn it into a web hook so you just code the parsing, and delivery is outsourced. Polling for changes is a terrible design. The source has to serve data on repeat to every reader that is set up to check once every week/day/hour/minute. At the very least have a distributed cache that is set to return only the entries created after generally agreed timestamps. (Please tell me that exists now, or why it’s a bad idea.)
  - aflukasz 2y ago
    > (Please tell me that exists now, or why it’s a bad idea.) Regarding the only-after semantics: - HTTP has "Delta encoding in HTTP": https://www.rfc-editor.org/rfc/rfc3229 https://www.rfc-editor.org/rfc/rfc3229 (2002), - usage for feeds proposed in 2004: https://bobwyman.typepad.com/main/2004/09/using_rfc3229_w.html https://bobwyman.typepad.com/main/2004/09/using_rfc3229_w.ht..., - some (not so) recent discussion (2017): https://www.ctrl.blog/entry/feed-delta-updates.html https://www.ctrl.blog/entry/feed-delta-updates.html + https://news.ycombinator.com/item?id=26091489 https://news.ycombinator.com/item?id=26091489 (zero comments). Adoption (or lack there of) is a different story. I guess the main issue is that it puts more work on the publisher side, in particular requires dynamic aspect when generating the response.
- prepend 2y ago
  I used yahoo pipes for this back in the day. I wanted to combine multiple feeds into a single date sort. So it gave me all the issues on like 20 repos from oldest to newest. This was much faster than clicking on 20 different feeds to see which ones are oldest or most recent.
Glench 2y ago
I needed something like this once but I forget why. Now I would probably just use val.town to build it instead of spinning up my own go server.
pavel_lishin 2y ago
Hah, I've recently had to solve the exact opposite problem - fetching a bunch of content from a single source, and splitting it into multiple feeds.
- bansheehash 2y ago
  Amen! The Android Developers blog (https://android-developers.googleblog.com https://android-developers.googleblog.com, available as RSS at https://android-developers.googleblog.com/atom.xml https://android-developers.googleblog.com/atom.xml) comes to mind - It includes updates on new compliance requirements to keep an app listed on Google Play, which I care about, mixed with a whole lot of crap I don't care about, like Google's latest shiny thing like Jetpack Compose, fluff pieces and marketing speak. How did you solve this problem?
  - pavel_lishin 2y ago
    Hand-written code :) I don't split an existing RSS feed, but I scrape a website, and store the posts in a database, with a column for the user who posted it. Then I generate an OPML file with URLs like http://localhost:1234?user= http://localhost:1234?user=... per user, and spin up a web server that fetches only that user's feed given the query parameter. Import the OPML into the news reader, and bingo bango.
  - pavo-etc 2y ago
    For my personal site I create RSS feeds for all the main posts, but also an RSS feed for each tag. If someone only wants to read my posts about TypeScript they can just follow the TypeScript tag instead of the main feed.
mococa 2y ago
I miss Yahoo Pipes so much.
- rcarmo 2y ago
  I use Node-RED for largely the same things these days, including aggregating and summarizing feeds with gpt35-turbo.
  - onli 2y ago
    Hey, that's interesting to me. Could you share a bit more about that? Do you replace the feeds with one summary item?
    - rcarmo 2y ago
      No, I collect all the items, save them to SQLite, and then batch them to Azure Open AI, generating a summary feed where each item is a set of summaries grouped by topic. One feed turns into 2-3 items a day in 8 hour windows.
- onli 2y ago
  I'm not claiming it's equivalent to Yahoo Pipes, but my https://pipes.digital https://pipes.digital does have a combine block ;)
tonyoconnell 2y ago
It will work well with RSS Proxy https://github.com/damoeb/rss-proxy https://github.com/damoeb/rss-proxy which turns any website into an RSS feed.
aranke 2y ago
Listen Notes lets you do this for podcasts: https://www.listennotes.com/blog/why-podcasts-are-my-new-wikipedia-the-perfect-41/ https://www.listennotes.com/blog/why-podcasts-are-my-new-wik...
ZunarJ5 2y ago
FreshRSS does this.
kevincox 2y ago
There is one footgun with many tools like this. If you add a new source to an existing combined feed all of the items in that feed will immediately appear as new items. What you typically want is for new items for the source feed to show up in the combined feed, not all exiting items at once.

61 points by 8organicbits 2y ago

I cannot see any use I'd have for that. All the RSS readers I've used in my life have supported reading from multiple feeds. Could someone enlighten me?

bertman 2y ago
I guess this moves complexity away from the client and enables a dead simple setup (maybe for non-technical friends and family?) with just a single big feed, like they already know from TikTok and the likes.
batch12 2y ago
Maybe this could be used to offer aggregated feeds as a service. It could be further refined to filter content based on criteria either provided by the user or predefined by the service. An example could be something like an infosec feed with curated sources and maybe an optional refinement for content related to vulnerabilities. This could also allow users to develop/fork the aggregated feeds and share them with a community.
- rambambram 2y ago
  This in combination with the <source> element from an item is a killer combo.
PeterStuer 2y ago
If you use rss as a middleware protocol, it saves you in client configuration. You deal with the updates and changes on the sources, for your client app it is just one never changing feed. Tbh, I run specialized vertical services aggregating 100's of sources (rss as well as other), and the combination towards a single feed is trivial.
crazygringo 2y ago
I am equally baffled. It's a pet peeve of mine seeing software projects that describe what they do, but without providing even a single use case. Or even what motivated the developer. Why does this thing exist? There is probably a reason and then I could learn something.
- sussmannbaka 2y ago
  A bunch of friends and I once toyed around with the idea of replacing microblogging through ActivityPub with one RSS feed per user. To then display them in one combined feed, I just added all of them to a category in my RSS reader, the way I already did with topics before. One of my friends' RSS reader didn’t have that option (grouping by category yes but no combined feed) so they instead combined the feeds into a single RSS feed and added that to their RSS reader. In the end it was kinda finicky so we went back to ActivityPub.
  - jayemar 2y ago
    I'm not sure if this is the functionality you are looking for, but Mastodon has built in RSS feeds per user. You just to add .rss to the user link. So, for example, if you had an account on mastodon.social, thered be an RSS feed of your posts at mastodon.social/@sussmannbaka.rss
    - trekz 2y ago
      Yeah, there are also third-party services that provide RSS feeds for additional Mastodon data like tags as well. e.g. https://openrss.org/mastodon.social/tags/assange https://openrss.org/mastodon.social/tags/assange
harryvederci 2y ago
I think koreader[0] has an RSS feature where you can fill in some RSS feeds and it converts feed items to epubs. It's nice, but I'm using a different reader on my laptop, so then syncing (of which feeds you follow) becomes a thing to do. Instead of following a bunch of them and having to sync them I guess you could just follow the combined one and be done with it. If you have multiple feed readers, each with their own way to input feeds, I think something like rsscombine might be an easy solution. [0] https://koreader.rocks/ https://koreader.rocks/
8organicbits 2y ago
Check out planets[1] as one use case. Projects like Debian, Mozilla, Python, and many more aggregate feeds into a planet to promote the blogs of members or other people who work in their space. Being able to subscribe to "Debian blogs" in one step is helpful versus discovering each blogger individually. There's a big list here [2], I think two use the linked code. [1] https://indieweb.org/planet https://indieweb.org/planet [2] https://github.com/ArchiveTeam/urls-sources/blob/master/43200_blog_aggregators.txt https://github.com/ArchiveTeam/urls-sources/blob/master/4320...
insom 2y ago
I saw this on HN and though “great I was just about to write this but now I don’t need to” - I have a legacy blog, a link blog and another thing I update which all produce RSS and I want to create one big feed of stuff I write, regardless of which platform I chose to put it on. Seems like I can use this for that.
- brokenmachine 2y ago
  But almost every RSS reader has this feature. I use FreshRSS and you just put all the feeds in one category and then you can click on that category to show all the feeds together.
  - saulpw 2y ago
    The reader has to add every RSS feed. What if I as the producer of those feeds want to ship a combined feed so they don't have to do that extra work? Or what if I'm a curator of RSS feeds and want to release one combined "low-tech computer news" feed sourced from 100 smaller low-tech computer news feeds?
    - rakoo 2y ago
      This is already solved with dynamic opml, which freshrss implements
richardw 2y ago
I’ve wondered if this existed, to simplify writing a system that parses and processes RSS. Frankly I’d go a bit further and spit out events, so people can code “give me the next entry” without having to mess with polling lots of rss endpoints. Just tell me when there’s a new entry. Or turn it into a web hook so you just code the parsing, and delivery is outsourced. Polling for changes is a terrible design. The source has to serve data on repeat to every reader that is set up to check once every week/day/hour/minute. At the very least have a distributed cache that is set to return only the entries created after generally agreed timestamps. (Please tell me that exists now, or why it’s a bad idea.)
- aflukasz 2y ago
  > (Please tell me that exists now, or why it’s a bad idea.) Regarding the only-after semantics: - HTTP has "Delta encoding in HTTP": https://www.rfc-editor.org/rfc/rfc3229 https://www.rfc-editor.org/rfc/rfc3229 (2002), - usage for feeds proposed in 2004: https://bobwyman.typepad.com/main/2004/09/using_rfc3229_w.html https://bobwyman.typepad.com/main/2004/09/using_rfc3229_w.ht..., - some (not so) recent discussion (2017): https://www.ctrl.blog/entry/feed-delta-updates.html https://www.ctrl.blog/entry/feed-delta-updates.html + https://news.ycombinator.com/item?id=26091489 https://news.ycombinator.com/item?id=26091489 (zero comments). Adoption (or lack there of) is a different story. I guess the main issue is that it puts more work on the publisher side, in particular requires dynamic aspect when generating the response.
prepend 2y ago
I used yahoo pipes for this back in the day. I wanted to combine multiple feeds into a single date sort. So it gave me all the issues on like 20 repos from oldest to newest. This was much faster than clicking on 20 different feeds to see which ones are oldest or most recent.

Glench 2y ago

I needed something like this once but I forget why. Now I would probably just use val.town to build it instead of spinning up my own go server.

pavel_lishin 2y ago

Hah, I've recently had to solve the exact opposite problem - fetching a bunch of content from a single source, and splitting it into multiple feeds.

bansheehash 2y ago
Amen! The Android Developers blog (https://android-developers.googleblog.com https://android-developers.googleblog.com, available as RSS at https://android-developers.googleblog.com/atom.xml https://android-developers.googleblog.com/atom.xml) comes to mind - It includes updates on new compliance requirements to keep an app listed on Google Play, which I care about, mixed with a whole lot of crap I don't care about, like Google's latest shiny thing like Jetpack Compose, fluff pieces and marketing speak. How did you solve this problem?
- pavel_lishin 2y ago
  Hand-written code :) I don't split an existing RSS feed, but I scrape a website, and store the posts in a database, with a column for the user who posted it. Then I generate an OPML file with URLs like http://localhost:1234?user= http://localhost:1234?user=... per user, and spin up a web server that fetches only that user's feed given the query parameter. Import the OPML into the news reader, and bingo bango.
- pavo-etc 2y ago
  For my personal site I create RSS feeds for all the main posts, but also an RSS feed for each tag. If someone only wants to read my posts about TypeScript they can just follow the TypeScript tag instead of the main feed.

mococa 2y ago

I miss Yahoo Pipes so much.

rcarmo 2y ago
I use Node-RED for largely the same things these days, including aggregating and summarizing feeds with gpt35-turbo.
- onli 2y ago
  Hey, that's interesting to me. Could you share a bit more about that? Do you replace the feeds with one summary item?
  - rcarmo 2y ago
    No, I collect all the items, save them to SQLite, and then batch them to Azure Open AI, generating a summary feed where each item is a set of summaries grouped by topic. One feed turns into 2-3 items a day in 8 hour windows.
onli 2y ago
I'm not claiming it's equivalent to Yahoo Pipes, but my https://pipes.digital https://pipes.digital does have a combine block ;)

tonyoconnell 2y ago

It will work well with RSS Proxy https://github.com/damoeb/rss-proxy https://github.com/damoeb/rss-proxy which turns any website into an RSS feed.

aranke 2y ago

Listen Notes lets you do this for podcasts: https://www.listennotes.com/blog/why-podcasts-are-my-new-wikipedia-the-perfect-41/ https://www.listennotes.com/blog/why-podcasts-are-my-new-wik...

ZunarJ5 2y ago

FreshRSS does this.

kevincox 2y ago

There is one footgun with many tools like this. If you add a new source to an existing combined feed all of the items in that feed will immediately appear as new items. What you typically want is for new items for the source feed to show up in the combined feed, not all exiting items at once.