8 ms·

RSSHub: Everything Is RSSible

214 points by hawkoy 6y ago

usrme 6y ago
Maybe someone else will benefit from my experience as well: I recently wanted to create an RSS feed for https://www.apmreports.org/ https://www.apmreports.org/ as they didn't seem to offer one, nor did Feedly (my preferred RSS reader) manage to find one that was up-to-date. I ended up using http://createfeed.fivefilters.org/ http://createfeed.fivefilters.org/ instead because there wasn't an existing route in RSSHub that I could use, and it didn't seem trivial to set one up for this particular case; I hope someone will prove me wrong though. With the Feed Creator feed URL I'm not able to see the content within Feedly (it just says "no content"), but it does relay new posts to me, which I can open directly, and it was amazingly quick to get a functional feed URL, so I'm very happy nonetheless.
pembrook 6y ago
Is there a list of available routes somewhere? I’m trying to create a personal clone of mailbrew to send myself a newsletter of all my favorite sources.
- xzyaoi 6y ago
  It's here. https://docs.rsshub.app/en/ https://docs.rsshub.app/en/ By the way, I've been using this for a while and it is superb.
- zmix 6y ago
  > Is there a list of available routes somewhere? This[1] gives you a JSON of all the routes. [1]: https://rsshub.app/api/routes https://rsshub.app/api/routes
dewey 6y ago
This is interesting, I was building something similar for my personal use case a while ago but there I have to build a plugin for each website I want to support which is not ideal (https://github.com/dewey/feedbridge https://github.com/dewey/feedbridge). I'll have to check it out, community maintained integrations make a lot more sense.
- cstuder 6y ago
  There is RSS-Bridge which is doing exactly that: A plugin per website, common libraries. A pragmatic approach. Trying to build a generalized framework handling each and every edge case promises endless complexity. https://github.com/RSS-Bridge/rss-bridge https://github.com/RSS-Bridge/rss-bridge
  - searchableguy 6y ago
    > Dear so-called "social" websites. > Your catchword is "share", but you don't want us to share. You want to keep us within your walled gardens. That's why you've been removing RSS links from webpages, hiding them deep on your website, or removed feeds entirely, replacing it with crippled or demented proprietary API. FUCK YOU. > You're not social when you hamper sharing by removing feeds. You're happy to have customers creating content for your ecosystem, but you don't want this content out - a content you do not even own. Google Takeout is just a gimmick. We want our data to flow, we want RSS or Atom feeds. > We want to share with friends, using open protocols: RSS, Atom, XMPP, whatever. Because no one wants to have your service with your applications using your API force-feeding them. Friends must be free to choose whatever software and service they want. > We are rebuilding bridges you have wilfully destroyed. Get your shit together: Put RSS/Atom back in. From that readme. Nice project! I wish someone would make a social rss reader. It would be pretty neat. Although, activitypub might be better protocol for that.
    - beagle3 6y ago
      Check out NewsBlur. It’s a social feed reader.
  - ThePadawan 6y ago
    I'm currently using rss-bridge and have not had that great of an experience. I put some of that down to the site-specific bridges, but some problems persist: - I regularly recieve feed updates that mark all articles as "new" (i.e. the bridge fails to recognize that it already exported some articles before) - I also often receive "error 521" from rss-bridge (i.e. the bridge cannot create a feed), but on manual investigation, the feed can be created just fine. I assume this is due to a lack of request throttling somewhere in the process, but can't be bothered to investigate.
chrismorgan 6y ago
I’m curious: why is this using RSS? RSS is uniformly technically inferior to Atom, and all feed reader apps that I know of support Atom, and libraries are available for both (some RSS-only, some Atom-only, some general). The only place I’d ever use RSS now is podcast feeds, because podcast feeds haven’t caught up up with 2007 yet.
- searchableguy 6y ago
  https://docs.rsshub.app/en/parameter.html#output-formats https://docs.rsshub.app/en/parameter.html#output-formats It does support atom. You need .atom at the end. By default, it's rss V2.
- erikrothoff 6y ago
  Most likely: RSS has better brand recognition than Atom feeds. There’s probably not a single RSS reader that doesn’t support Atom too.
  - avian 6y ago
    RSS is basically a synonym for Atom at this point.
- onli 6y ago
  There is no reason to use Atom. RSS has in practice some specifics like content:encoded and description that are strange at first look, but by now are solved in every reader. So why not just use the standard that works?
  - bmn__ 6y ago
    As someone who has done this: it's not solved in every reader, and the standards do not work. Atom is interoperable and extensible and great care was taken during design and implementation testing to achieve these goals. RSS, OTOH, is garbage on fire ruined by one man's ego. https://web.archive.org/web/2004/http://diveintomark.org/archives/2004/02/04/incompatible-rss https://web.archive.org/web/2004/http://diveintomark.org/arc...
rozab 6y ago
I have genuinely found that open source projects by people with anime girls as their profile pic tend to be excellent
- blindm 6y ago
  Also if you're interested, see how many anime avatars are present in the contributors chart: https://camo.githubusercontent.com/665e81f281b143b7da39097165d7a141b08f7860/68747470733a2f2f6f70656e636f6c6c6563746976652e636f6d2f5253534875622f636f6e7472696275746f72732e7376673f77696474683d383930 https://camo.githubusercontent.com/665e81f281b143b7da3909716...
- gilrain 6y ago
  Those may be correlated with a lack of family and, thus, more free time for personal projects. Not that one outgrows an appreciation of anime once married, but that the tendency toward cute girl avatars probably decreases.
  - prophesi 6y ago
    I was going to counter that it could instead just be that they're not employed, in education, or in training. But personally, I'm employed and can devote entire weekends and evenings to side projects. Family would certainly take up the rest of that time. Edit: Actually just noticed your comment was about anime girl avatars, and not the project quality. I have no opinions on that matter.
  - deleted 6y ago
    [deleted]
- blindm 6y ago
  You mean this avatar? https://avatars3.githubusercontent.com/u/8266075?s=460&u=60d5e8f8f6f83bd76542295c5112b3096fdb7d0d&v=4 https://avatars3.githubusercontent.com/u/8266075?s=460&u=60d...
phre4k 6y ago
I just love that Asian developers finally 'discovered' the English speaking FOSS community. A lot of the software I use and contribute to has gained many contributors from China, Singapore and Japan through the last 1-2 years. It really warms my heart to see that FOSS knows less and less borders :)
- mindfulhack 6y ago
  It especially does because people think that being against 'Chinese apps' like TikTok is 'sinophobia', when really we love Chinese people it's just the government we don't trust. So it really does makes it special to see those Chinese characters in that README. I want to see more of that. But I also immediately feel concern. I hope this Shanghai developer is not targeted, bullied, or persecuted in any way like having their money seized. Damn. Just donated to their PayPal because I feel so encouraged by it.
  - ddorian43 6y ago
    I always see this, we love the people but not the government. Is there a case where we hate the people ? Are people as a collective harmless.l ?
    - phre4k 6y ago
      Racists exist. They hate people.
      - ddorian43 6y ago
        I'm not talking about races. Mostly big groups, like countries, which don't have to be races.
  - altindag 6y ago
    I think this is bullshit. It's like saying, "I'm not racist, my friend Bob is black." You don't get a "get outta jail free" card for saying "I love CHinese people" and "oorah for China devs" and then heaping shit all over their country's achievements the rest of the time. Do people who say this think they understand China or Chinese people even a little bit?
genmon 6y ago
I'm pleased to see renewed interest in RSS (web feeds generally). But my concern is that, for first time users, the onboarding process is too complex. There are too many unexplained steps, and the benefits are unclear. A plug for my own answer to this: I recently launched https://aboutfeeds.com https://aboutfeeds.com -- a one pager "getting started" guide aimed at first-time users of web feeds/RSS. The goal is to have it linked next to every "RSS subscribe" button out there.
- boastful_inaba 6y ago
  Hey, looks like you had a very similar idea to me! https://www.youneedfeeds.com/ https://www.youneedfeeds.com/ Though you've gone for the 'simple one-pager' approach where I tried to sell people on it via a full site. I was going to do a visual revamp sometime soon, hopefully include more readers etc but time has been an issue.
  - genmon 6y ago
    This is great! I especially love the starter packs. We both opted for "web feeds" over "RSS", which makes me feel better about that choice. Do you have any sense of whether you've been able to reach new users? It's going to take a lot of effort from us and people like us, I think.
    - boastful_inaba 6y ago
      I chose 'web feeds' since it covered both RSS/ATOM and avoided the need to define their acronym meaning. My view stats were very low last time I checked, but I haven't done much in the way of promotion either. (I did look into keyword advertising but that was prohibitively expensive.) I'll probably try for another round of organic promotion once I do the site overhaul.
  - Can_Not 6y ago
    But neither of your websites have a section for website authors!
- 6y ago
monkeydust 6y ago
What might be the best way to aggregate multiple RSS feeds onto a webpage? So have been recently thinking about density of information and I want to create a webpage with as much information as possible and park it on a large 55inch screen I have in my home-office.
acidburnNSA 6y ago
Nice! I am excited by this. I decided to go back to trying RSS after a blogger I enjoy got deplatformed from where I usually found their stuff. I installed a firefox rss reader and it was fine. But today after seeing this I realized I wanted more centralized RSS to have it synced on difference devices. So I went and installed FreshRSS over the past 30 mins on my webserver, added a MariaSQL db and user, added a Apache subdomain, directed my DNS to it, ran letsencrypt, fired up the web config, added my feeds, and we're up and running in 30 mins flat. Next I guess I have to install this thing to RSS-ify other channels I like. This is kind of exciting.
rmetzler 6y ago
Someone at my company wanted to use RSShub for turning Instagram into an RSS feed. This kept breaking, because rsshub just scrapes Instagram profiles on the web and Instagram pushes its users to the app. It’s probably more Instagrams fault than rsshub’s that it was breaking. Just be aware of these limitations.
- edsimpson 6y ago
  No affiliation, but check out InstaFeed. Allows you to authenticate and generate an rss feed for an Instagram profile. https://github.com/falzm/instafeed https://github.com/falzm/instafeed
- IceWreck 6y ago
  RSSBridge does something similar, and when instagram breaks, its usually in two -three days
thedz 6y ago
I've started using IFTTT to sent RSS feed updates over to my email, and I think I've finally found my preferred way of consuming feeds. My email client is actually much more powerful than most readers in letting me capture read status and labels / sorting.
- Seirdy 6y ago
  You might be interested in feed2maildir [1], which saves feeds in maildir format so you can use all your favorite mail clients and index with notmuch. That way, you aren't dependent on another centralized privacy-invasive proprietary SaaS walled garden being online and running without major changes. I've been planning on transitioning from my current TUI RSS reader (Newsboat) to a maildir-based CLI with notmuch filters for a while; I might give it a shot next weekend and post some of my notmuch filters for RSS. [1]: https://github.com/sulami/feed2maildir https://github.com/sulami/feed2maildir
WrongThinkerNo5 6y ago
Am I missing how RSSHub is a feed aggregator? It seems to be a feed generator that works with an aggregator/reader.
jokz 6y ago
if privacy matter for use use this https://privacytoolslist.com/#feed-reader https://privacytoolslist.com/#feed-reader open source rss readers and tools to manage your feeds
laike9m 6y ago
I've been using RSSHub and it's GREAT. It surprises me that people here fount it out so late. The project has been quite popular for years.

214 points by hawkoy 6y ago

Maybe someone else will benefit from my experience as well: I recently wanted to create an RSS feed for https://www.apmreports.org/ https://www.apmreports.org/ as they didn't seem to offer one, nor did Feedly (my preferred RSS reader) manage to find one that was up-to-date. I ended up using http://createfeed.fivefilters.org/ http://createfeed.fivefilters.org/ instead because there wasn't an existing route in RSSHub that I could use, and it didn't seem trivial to set one up for this particular case; I hope someone will prove me wrong though. With the Feed Creator feed URL I'm not able to see the content within Feedly (it just says "no content"), but it does relay new posts to me, which I can open directly, and it was amazingly quick to get a functional feed URL, so I'm very happy nonetheless.

pembrook 6y ago

Is there a list of available routes somewhere? I’m trying to create a personal clone of mailbrew to send myself a newsletter of all my favorite sources.

xzyaoi 6y ago
It's here. https://docs.rsshub.app/en/ https://docs.rsshub.app/en/ By the way, I've been using this for a while and it is superb.
zmix 6y ago
> Is there a list of available routes somewhere? This[1] gives you a JSON of all the routes. [1]: https://rsshub.app/api/routes https://rsshub.app/api/routes

dewey 6y ago

This is interesting, I was building something similar for my personal use case a while ago but there I have to build a plugin for each website I want to support which is not ideal (https://github.com/dewey/feedbridge https://github.com/dewey/feedbridge). I'll have to check it out, community maintained integrations make a lot more sense.

cstuder 6y ago
There is RSS-Bridge which is doing exactly that: A plugin per website, common libraries. A pragmatic approach. Trying to build a generalized framework handling each and every edge case promises endless complexity. https://github.com/RSS-Bridge/rss-bridge https://github.com/RSS-Bridge/rss-bridge
- searchableguy 6y ago
  > Dear so-called "social" websites. > Your catchword is "share", but you don't want us to share. You want to keep us within your walled gardens. That's why you've been removing RSS links from webpages, hiding them deep on your website, or removed feeds entirely, replacing it with crippled or demented proprietary API. FUCK YOU. > You're not social when you hamper sharing by removing feeds. You're happy to have customers creating content for your ecosystem, but you don't want this content out - a content you do not even own. Google Takeout is just a gimmick. We want our data to flow, we want RSS or Atom feeds. > We want to share with friends, using open protocols: RSS, Atom, XMPP, whatever. Because no one wants to have your service with your applications using your API force-feeding them. Friends must be free to choose whatever software and service they want. > We are rebuilding bridges you have wilfully destroyed. Get your shit together: Put RSS/Atom back in. From that readme. Nice project! I wish someone would make a social rss reader. It would be pretty neat. Although, activitypub might be better protocol for that.
  - beagle3 6y ago
    Check out NewsBlur. It’s a social feed reader.
- ThePadawan 6y ago
  I'm currently using rss-bridge and have not had that great of an experience. I put some of that down to the site-specific bridges, but some problems persist: - I regularly recieve feed updates that mark all articles as "new" (i.e. the bridge fails to recognize that it already exported some articles before) - I also often receive "error 521" from rss-bridge (i.e. the bridge cannot create a feed), but on manual investigation, the feed can be created just fine. I assume this is due to a lack of request throttling somewhere in the process, but can't be bothered to investigate.

chrismorgan 6y ago

I’m curious: why is this using RSS? RSS is uniformly technically inferior to Atom, and all feed reader apps that I know of support Atom, and libraries are available for both (some RSS-only, some Atom-only, some general). The only place I’d ever use RSS now is podcast feeds, because podcast feeds haven’t caught up up with 2007 yet.

searchableguy 6y ago
https://docs.rsshub.app/en/parameter.html#output-formats https://docs.rsshub.app/en/parameter.html#output-formats It does support atom. You need .atom at the end. By default, it's rss V2.
erikrothoff 6y ago
Most likely: RSS has better brand recognition than Atom feeds. There’s probably not a single RSS reader that doesn’t support Atom too.
- avian 6y ago
  RSS is basically a synonym for Atom at this point.
onli 6y ago
There is no reason to use Atom. RSS has in practice some specifics like content:encoded and description that are strange at first look, but by now are solved in every reader. So why not just use the standard that works?
- bmn__ 6y ago
  As someone who has done this: it's not solved in every reader, and the standards do not work. Atom is interoperable and extensible and great care was taken during design and implementation testing to achieve these goals. RSS, OTOH, is garbage on fire ruined by one man's ego. https://web.archive.org/web/2004/http://diveintomark.org/archives/2004/02/04/incompatible-rss https://web.archive.org/web/2004/http://diveintomark.org/arc...

rozab 6y ago

I have genuinely found that open source projects by people with anime girls as their profile pic tend to be excellent

blindm 6y ago
Also if you're interested, see how many anime avatars are present in the contributors chart: https://camo.githubusercontent.com/665e81f281b143b7da39097165d7a141b08f7860/68747470733a2f2f6f70656e636f6c6c6563746976652e636f6d2f5253534875622f636f6e7472696275746f72732e7376673f77696474683d383930 https://camo.githubusercontent.com/665e81f281b143b7da3909716...
gilrain 6y ago
Those may be correlated with a lack of family and, thus, more free time for personal projects. Not that one outgrows an appreciation of anime once married, but that the tendency toward cute girl avatars probably decreases.
- prophesi 6y ago
  I was going to counter that it could instead just be that they're not employed, in education, or in training. But personally, I'm employed and can devote entire weekends and evenings to side projects. Family would certainly take up the rest of that time. Edit: Actually just noticed your comment was about anime girl avatars, and not the project quality. I have no opinions on that matter.
- deleted 6y ago
  [deleted]
blindm 6y ago
You mean this avatar? https://avatars3.githubusercontent.com/u/8266075?s=460&u=60d5e8f8f6f83bd76542295c5112b3096fdb7d0d&v=4 https://avatars3.githubusercontent.com/u/8266075?s=460&u=60d...

phre4k 6y ago

I just love that Asian developers finally 'discovered' the English speaking FOSS community. A lot of the software I use and contribute to has gained many contributors from China, Singapore and Japan through the last 1-2 years. It really warms my heart to see that FOSS knows less and less borders :)

mindfulhack 6y ago
It especially does because people think that being against 'Chinese apps' like TikTok is 'sinophobia', when really we love Chinese people it's just the government we don't trust. So it really does makes it special to see those Chinese characters in that README. I want to see more of that. But I also immediately feel concern. I hope this Shanghai developer is not targeted, bullied, or persecuted in any way like having their money seized. Damn. Just donated to their PayPal because I feel so encouraged by it.
- ddorian43 6y ago
  I always see this, we love the people but not the government. Is there a case where we hate the people ? Are people as a collective harmless.l ?
  - phre4k 6y ago
    Racists exist. They hate people.
    - ddorian43 6y ago
      I'm not talking about races. Mostly big groups, like countries, which don't have to be races.
- altindag 6y ago
  I think this is bullshit. It's like saying, "I'm not racist, my friend Bob is black." You don't get a "get outta jail free" card for saying "I love CHinese people" and "oorah for China devs" and then heaping shit all over their country's achievements the rest of the time. Do people who say this think they understand China or Chinese people even a little bit?

genmon 6y ago

I'm pleased to see renewed interest in RSS (web feeds generally). But my concern is that, for first time users, the onboarding process is too complex. There are too many unexplained steps, and the benefits are unclear. A plug for my own answer to this: I recently launched https://aboutfeeds.com https://aboutfeeds.com -- a one pager "getting started" guide aimed at first-time users of web feeds/RSS. The goal is to have it linked next to every "RSS subscribe" button out there.

boastful_inaba 6y ago
Hey, looks like you had a very similar idea to me! https://www.youneedfeeds.com/ https://www.youneedfeeds.com/ Though you've gone for the 'simple one-pager' approach where I tried to sell people on it via a full site. I was going to do a visual revamp sometime soon, hopefully include more readers etc but time has been an issue.
- genmon 6y ago
  This is great! I especially love the starter packs. We both opted for "web feeds" over "RSS", which makes me feel better about that choice. Do you have any sense of whether you've been able to reach new users? It's going to take a lot of effort from us and people like us, I think.
  - boastful_inaba 6y ago
    I chose 'web feeds' since it covered both RSS/ATOM and avoided the need to define their acronym meaning. My view stats were very low last time I checked, but I haven't done much in the way of promotion either. (I did look into keyword advertising but that was prohibitively expensive.) I'll probably try for another round of organic promotion once I do the site overhaul.
- Can_Not 6y ago
  But neither of your websites have a section for website authors!
6y ago

monkeydust 6y ago

What might be the best way to aggregate multiple RSS feeds onto a webpage? So have been recently thinking about density of information and I want to create a webpage with as much information as possible and park it on a large 55inch screen I have in my home-office.

acidburnNSA 6y ago

Nice! I am excited by this. I decided to go back to trying RSS after a blogger I enjoy got deplatformed from where I usually found their stuff. I installed a firefox rss reader and it was fine. But today after seeing this I realized I wanted more centralized RSS to have it synced on difference devices. So I went and installed FreshRSS over the past 30 mins on my webserver, added a MariaSQL db and user, added a Apache subdomain, directed my DNS to it, ran letsencrypt, fired up the web config, added my feeds, and we're up and running in 30 mins flat. Next I guess I have to install this thing to RSS-ify other channels I like. This is kind of exciting.

rmetzler 6y ago

Someone at my company wanted to use RSShub for turning Instagram into an RSS feed. This kept breaking, because rsshub just scrapes Instagram profiles on the web and Instagram pushes its users to the app. It’s probably more Instagrams fault than rsshub’s that it was breaking. Just be aware of these limitations.

edsimpson 6y ago
No affiliation, but check out InstaFeed. Allows you to authenticate and generate an rss feed for an Instagram profile. https://github.com/falzm/instafeed https://github.com/falzm/instafeed
IceWreck 6y ago
RSSBridge does something similar, and when instagram breaks, its usually in two -three days

thedz 6y ago

I've started using IFTTT to sent RSS feed updates over to my email, and I think I've finally found my preferred way of consuming feeds. My email client is actually much more powerful than most readers in letting me capture read status and labels / sorting.

Seirdy 6y ago
You might be interested in feed2maildir [1], which saves feeds in maildir format so you can use all your favorite mail clients and index with notmuch. That way, you aren't dependent on another centralized privacy-invasive proprietary SaaS walled garden being online and running without major changes. I've been planning on transitioning from my current TUI RSS reader (Newsboat) to a maildir-based CLI with notmuch filters for a while; I might give it a shot next weekend and post some of my notmuch filters for RSS. [1]: https://github.com/sulami/feed2maildir https://github.com/sulami/feed2maildir

WrongThinkerNo5 6y ago

Am I missing how RSSHub is a feed aggregator? It seems to be a feed generator that works with an aggregator/reader.

jokz 6y ago

if privacy matter for use use this https://privacytoolslist.com/#feed-reader https://privacytoolslist.com/#feed-reader open source rss readers and tools to manage your feeds

laike9m 6y ago

I've been using RSSHub and it's GREAT. It surprises me that people here fount it out so late. The project has been quite popular for years.