7 ms·

Deep dive into finding RSS feeds

144 points by domysee 2y ago

kelvinjps10 2y ago
I would like to be able put multiple websites, I had to build a script based on "Guessing the feed URL" approach to get the rss feeed of a bunch of websites that I had bookmarked
superkuh 2y ago
I generally try: /rss, /feed, /index.xml, /rss.xml, /feed.xml, etc. And at various root or /directory/* locations. https://blog.jim-nielsen.com/2021/feed-urls/ https://blog.jim-nielsen.com/2021/feed-urls/ is a good article with statistics on naming. I've been adding to my feeds.opml since reddit started dying in ~2015 and now I'm up to around ~1700 feeds and mostly independent from aggregators; though I still collect new feeds from HN/IRC/etc. Mostly I just always make a point to look for them whenever I read something cool on the web.
- yazmeya 2y ago
  You should also check if the web page actually exposes this information in a <link rel="alternate"> tag. If you're running Chrome, the "RSS Subscription Extension (by Google)" extension [1] will do this for you automatically and light up an orange icon in the extensions bar. It also integrates with popular RSS aggregators so you can subscribe directly from the extension. [1] https://chromewebstore.google.com/detail/rss-subscription-extensio/nlbjncdgjeocebhnmkbbbdekmmmcbfjd?pli=1 https://chromewebstore.google.com/detail/rss-subscription-ex...
  - superkuh 2y ago
    My browser is a fork of Firefox made for web surfing (as opposed to running javascript applications). It never removed that functionality like FF/Chrome/etc did. Such auto-discovery is included in base Palemoon as it use to be in base Firefox.
ulrischa 2y ago
Would be nice if it is implmented in freshrss
- quaintdev 2y ago
  I think miniflux can do this. I just give it website address and it almost always finds rss feeds
1123581321 2y ago
It’d be neat for readers to seamlessly integrate with a scraper, either self-hosted or commercial, if no feed is found. I believe Inoreader allows scraping a few sites depending on the plan level; most reader services don’t.
HumblyTossed 2y ago
Back when I was young, websites had this icon you could click that would take you straight to their RSS feed. You young whipper snappers have gone an fucked that up. Actually, I think it was Google's fault. When they killed their RSS reader people pronounced RSS dead so people just stopped publishing RSS feeds or just didn't link to them. * Yes, I know the article talks about the RSS icon, i'm just soapboxing.
- AndrewStephens 2y ago
  Even better, for a few months the browsers themselves would highlight RSS feeds and allow you subscribe right in the browser. It was too good to last. RSS is great but it has one great flaw in that it doesn't scale that well by itself. If 2 million people subscribe to your feed and try to update it once an hour, that is 48 million requests a day just for RSS. What does work well (and how things have evolved) is to have a service that polls RSS on behalf of its users. This was the beauty of Google Reader but plenty of replacements exist.
  - rodary 2y ago
    > Even better, for a few months the browsers themselves would highlight RSS feeds and allow you subscribe right in the browser. It was too good to last. Vivaldi browser still does that.
  - qw 2y ago
    > RSS is great but it has one great flaw in that it doesn't scale that well by itself The readers often cache the result for all users of the same feed. The only exception are readers that are running 100% locally. In that case using Etag or Last-Modified will make each request cheap and manageble.
    - AndrewStephens 2y ago
      That is my point. RSS by itself is a scaling problem but online readers make it manageable by polling sites on a reasonable rate and redistributing new articles to their users.
sodality2 2y ago
Tried out the feed finder on my blog again and I have another bug to report - it seems the URLs on the page can cause a crash within the web app! my blog (at matthew.science) uses Zola SSG, and it seems the URLs are formatted with a preceding //: '<a href="//matthew.science/posts/riscv/">Basics of the RISC-V ISA</a>' This causes the following error: TypeError: URL constructor: //matthew.science/posts/riscv/ is not a valid URL.
- jdougan 2y ago
  Theoretically, I think that should work. It (At least it used to be) specified that //site/some/path should assume the uri protocol of the current context. So if it was a link on an http page, it should assume http, same with ftp and https etc. It should work sorta like how a leading slash assumes the current site context. This was back before the Web became the one true way and is the reason it uses 2 slashes, to distinguish protocol local from site local.
- domysee 2y ago
  Thank you! Didn't see that case yet but I'm glad you commented, now I can fix it
  - domysee 2y ago
    Update: it's fixed
camel-cdr 2y ago
This is useful, I set up RSS on my website yesterday. Turns out the feed finder couldn't find the feeds even though I've linked to them using clickable RSS icons. I didn't know about the autodiscovery feature so I'll add that now.
zenlot 2y ago
Came here through RSS link from miniflux, running on nvidia jetson.
LorenDB 2y ago
My modus operandi for finding a non-obvious RSS feed is to check the Wayback Machine's list of saved URLs and search for "RSS", "feed", or "XML". That normally will find the feed as long as it exists.
saaaaaam 2y ago
Ghost also publishes at /feed
deleted 2y ago
[deleted]
csswizardry 2y ago
I went canvassing for RSS feeds only yesterday! Some good stuff in here: https://bsky.app/profile/csswizardry.com/post/3lckq4qo6zs22 https://bsky.app/profile/csswizardry.com/post/3lckq4qo6zs22
artembugara 2y ago
I open-sourced pyGoogleNews and wrote a quick blog about how you can reverse engineer google news RSS to turn it into an RSS feed of any website that is supported by Google News https://news.ycombinator.com/item?id=42343182 https://news.ycombinator.com/item?id=42343182 https://github.com/kotartemiy/pygooglenews https://github.com/kotartemiy/pygooglenews
ks2048 2y ago
It would be nice if someone ran this on commoncrawl and published a list of all the RSS feeds. (probably someone has?) Or I suppose you could just find all "Content-type: application/rss+xml" in CC. I know in the past, when I was looking for large lists of RSS feeds, I didn't really find what I was looking for.
- freetonik 2y ago
  Far from a complete list, but I maintain a curated list of personal blogs with RSS feeds here https://minifeed.net/blogs https://minifeed.net/blogs
X--corner--X 2y ago
[flagged]
benrapscallion 2y ago
Does it correctly ignore the “Comments on:” feeds that are sometimes mistakenly chosen over the main feed?
- domysee 2y ago
  It shows all feeds it finds, including comments feeds. It also shows a preview of feed content, so it's easy to see if it's only articles or also comments.
begriffs 2y ago
I created a lightweight shell script to check many url combinations on a site for feeds. https://github.com/begriffs/findrss https://github.com/begriffs/findrss The combinations came from what I observed in the big list of blogs I follow. The script works pretty well for most sites.
panozzaj 2y ago
I use a Chrome extension (https://chromewebstore.google.com/detail/get-rss-feed-url/kfghpdldaipanmkhfpdcjglncmilendn?hl=en&pli=1 https://chromewebstore.google.com/detail/get-rss-feed-url/kf...) and it seems to pick out the RSS URLs fairly consistently
fallinditch 2y ago
This looks very useful. It would work well with Hoarder (would be cool if they were integrated ;) Note: Hoarder can automatically hoard RSS feeds as part of its 'bookmark everything' functionality. Hoarder uses AI to tag all the content (URLs, feeds, images, notes) so you can then do full text searches on your personal archive of your bookmarks etc. https://hoarder.app/ https://hoarder.app/
- djhn 2y ago
  Awesome piece of software and right up my alley (although I’m too invested in my homegrown scripts to let them go - but I’ll give this a try.) You might consider adding some of the key info from the first paragraphs in the docs (open source, self hosting) to the front page, above the fold. Github-link might imply it, but I was scanning for “open source” with my eyes and was initially disappointed not to find it and ready to dismiss the product right out of the gate.
  - fallinditch 2y ago
    I didn't make this software, but heard about it on another thread. I was saying 'there should be an app like an 'auto-RAG' that scrapes RSS feeds and URLs' and be_erik wrote 'I do exactly this with hoarder. I passively build tagged knowledge bases with the archived pages and then feed it to a RAG setup.' What does your setup look like?
    - djhn 2y ago
      Things I bookmark (browsers), like and save (youtube, tiktok), upvote (hn) get archived to both internet archive and my own storage.
PeterStuer 2y ago
You can add '/display-feed.rss' to the list of common suffixes for many .eu sites
- domysee 2y ago
  Thank you - will do!
aucisson_masque 2y ago
I thank WordPress for most of my RSS feed. I follow mostly RSS on non technology website, for instance road cycling. people that wouldn't care or know about RSS because they are not very techy, yet because they are normies that use WordPress for all their website it puts a page with RSS feed automatically. You got to find it with developer tool by searching RSS but 99% of the time if it's WordPress it got RSS. Thank you WordPress you bloated piece of shit :)
openrisk 2y ago
Its interesting to contemplate an RSS-first browser that would have this functionality built-in. Think for example of promoting to full browser status a desktop RSS reader like Akregator [1] (which already embeds a webview). The browser as we now know it is mostly a static application that has long lost its user-centric mission. Websites might push some stuff but the user must do thinks manually. Its primary function is to provide a search window to external search. People even stopped using bookmarks and search for everything. This hypothetical RSS-Browser could become the main organizational tool for the users web experience, integrating the use of bookmarks. In fact even more "feeds" could be integrated like email and activitypub or atproto posts. It boils down to the fact that each person has a number of profiles/roles and within each they have a taxonomy of interests and we need a tool that integrates static and dynamic sources of information. [1] https://apps.kde.org/akregator/ https://apps.kde.org/akregator/
renegat0x0 2y ago
I fought this problem, since I wrote my own RSS reader in python. Might not be perfect. The problem with the approach presented here is speed. Most of the web pages, especially smaller are really slow. Crawling most of the web pages is pain, especially if you use selenium and small SBC. Therefore either the page presents a clean nice RSS link, or get lost. Most of the good, modern pages give you nice RSS. Even GitHub gives you RSS for commits. For other pages I try openRSS. For YouTube I use yt-dlp to obtain channel id, to establish RSS. Algorithm is crude, but gets the job done. https://github.com/rumca-js/Django-link-archive/blob/main/rsshistory/webtools/url.py https://github.com/rumca-js/Django-link-archive/blob/main/rs...
ewired 2y ago
Pasting a URL in NewsBlur also uses several of these techniques to find the feed(s), and it is open source, so the feed-finding code could be ripped out of NewsBlur as an alternative to this.

144 points by domysee 2y ago

I would like to be able put multiple websites, I had to build a script based on "Guessing the feed URL" approach to get the rss feeed of a bunch of websites that I had bookmarked

superkuh 2y ago

I generally try: /rss, /feed, /index.xml, /rss.xml, /feed.xml, etc. And at various root or /directory/* locations. https://blog.jim-nielsen.com/2021/feed-urls/ https://blog.jim-nielsen.com/2021/feed-urls/ is a good article with statistics on naming. I've been adding to my feeds.opml since reddit started dying in ~2015 and now I'm up to around ~1700 feeds and mostly independent from aggregators; though I still collect new feeds from HN/IRC/etc. Mostly I just always make a point to look for them whenever I read something cool on the web.

yazmeya 2y ago
You should also check if the web page actually exposes this information in a <link rel="alternate"> tag. If you're running Chrome, the "RSS Subscription Extension (by Google)" extension [1] will do this for you automatically and light up an orange icon in the extensions bar. It also integrates with popular RSS aggregators so you can subscribe directly from the extension. [1] https://chromewebstore.google.com/detail/rss-subscription-extensio/nlbjncdgjeocebhnmkbbbdekmmmcbfjd?pli=1 https://chromewebstore.google.com/detail/rss-subscription-ex...
- superkuh 2y ago
  My browser is a fork of Firefox made for web surfing (as opposed to running javascript applications). It never removed that functionality like FF/Chrome/etc did. Such auto-discovery is included in base Palemoon as it use to be in base Firefox.

ulrischa 2y ago

Would be nice if it is implmented in freshrss

quaintdev 2y ago
I think miniflux can do this. I just give it website address and it almost always finds rss feeds

1123581321 2y ago

It’d be neat for readers to seamlessly integrate with a scraper, either self-hosted or commercial, if no feed is found. I believe Inoreader allows scraping a few sites depending on the plan level; most reader services don’t.

HumblyTossed 2y ago

Back when I was young, websites had this icon you could click that would take you straight to their RSS feed. You young whipper snappers have gone an fucked that up. Actually, I think it was Google's fault. When they killed their RSS reader people pronounced RSS dead so people just stopped publishing RSS feeds or just didn't link to them. * Yes, I know the article talks about the RSS icon, i'm just soapboxing.

AndrewStephens 2y ago
Even better, for a few months the browsers themselves would highlight RSS feeds and allow you subscribe right in the browser. It was too good to last. RSS is great but it has one great flaw in that it doesn't scale that well by itself. If 2 million people subscribe to your feed and try to update it once an hour, that is 48 million requests a day just for RSS. What does work well (and how things have evolved) is to have a service that polls RSS on behalf of its users. This was the beauty of Google Reader but plenty of replacements exist.
- rodary 2y ago
  > Even better, for a few months the browsers themselves would highlight RSS feeds and allow you subscribe right in the browser. It was too good to last. Vivaldi browser still does that.
- qw 2y ago
  > RSS is great but it has one great flaw in that it doesn't scale that well by itself The readers often cache the result for all users of the same feed. The only exception are readers that are running 100% locally. In that case using Etag or Last-Modified will make each request cheap and manageble.
  - AndrewStephens 2y ago
    That is my point. RSS by itself is a scaling problem but online readers make it manageable by polling sites on a reasonable rate and redistributing new articles to their users.

sodality2 2y ago

Tried out the feed finder on my blog again and I have another bug to report - it seems the URLs on the page can cause a crash within the web app! my blog (at matthew.science) uses Zola SSG, and it seems the URLs are formatted with a preceding //: '<a href="//matthew.science/posts/riscv/">Basics of the RISC-V ISA</a>' This causes the following error: TypeError: URL constructor: //matthew.science/posts/riscv/ is not a valid URL.

jdougan 2y ago
Theoretically, I think that should work. It (At least it used to be) specified that //site/some/path should assume the uri protocol of the current context. So if it was a link on an http page, it should assume http, same with ftp and https etc. It should work sorta like how a leading slash assumes the current site context. This was back before the Web became the one true way and is the reason it uses 2 slashes, to distinguish protocol local from site local.
domysee 2y ago
Thank you! Didn't see that case yet but I'm glad you commented, now I can fix it
- domysee 2y ago
  Update: it's fixed

camel-cdr 2y ago

This is useful, I set up RSS on my website yesterday. Turns out the feed finder couldn't find the feeds even though I've linked to them using clickable RSS icons. I didn't know about the autodiscovery feature so I'll add that now.

zenlot 2y ago

Came here through RSS link from miniflux, running on nvidia jetson.

LorenDB 2y ago

My modus operandi for finding a non-obvious RSS feed is to check the Wayback Machine's list of saved URLs and search for "RSS", "feed", or "XML". That normally will find the feed as long as it exists.

saaaaaam 2y ago

Ghost also publishes at /feed

deleted 2y ago

[deleted]

csswizardry 2y ago

I went canvassing for RSS feeds only yesterday! Some good stuff in here: https://bsky.app/profile/csswizardry.com/post/3lckq4qo6zs22 https://bsky.app/profile/csswizardry.com/post/3lckq4qo6zs22

artembugara 2y ago

I open-sourced pyGoogleNews and wrote a quick blog about how you can reverse engineer google news RSS to turn it into an RSS feed of any website that is supported by Google News https://news.ycombinator.com/item?id=42343182 https://news.ycombinator.com/item?id=42343182 https://github.com/kotartemiy/pygooglenews https://github.com/kotartemiy/pygooglenews

ks2048 2y ago

It would be nice if someone ran this on commoncrawl and published a list of all the RSS feeds. (probably someone has?) Or I suppose you could just find all "Content-type: application/rss+xml" in CC. I know in the past, when I was looking for large lists of RSS feeds, I didn't really find what I was looking for.

freetonik 2y ago
Far from a complete list, but I maintain a curated list of personal blogs with RSS feeds here https://minifeed.net/blogs https://minifeed.net/blogs

X--corner--X 2y ago

[flagged]

benrapscallion 2y ago

Does it correctly ignore the “Comments on:” feeds that are sometimes mistakenly chosen over the main feed?

domysee 2y ago
It shows all feeds it finds, including comments feeds. It also shows a preview of feed content, so it's easy to see if it's only articles or also comments.

begriffs 2y ago

I created a lightweight shell script to check many url combinations on a site for feeds. https://github.com/begriffs/findrss https://github.com/begriffs/findrss The combinations came from what I observed in the big list of blogs I follow. The script works pretty well for most sites.

panozzaj 2y ago

I use a Chrome extension (https://chromewebstore.google.com/detail/get-rss-feed-url/kfghpdldaipanmkhfpdcjglncmilendn?hl=en&pli=1 https://chromewebstore.google.com/detail/get-rss-feed-url/kf...) and it seems to pick out the RSS URLs fairly consistently

fallinditch 2y ago

This looks very useful. It would work well with Hoarder (would be cool if they were integrated ;) Note: Hoarder can automatically hoard RSS feeds as part of its 'bookmark everything' functionality. Hoarder uses AI to tag all the content (URLs, feeds, images, notes) so you can then do full text searches on your personal archive of your bookmarks etc. https://hoarder.app/ https://hoarder.app/

djhn 2y ago
Awesome piece of software and right up my alley (although I’m too invested in my homegrown scripts to let them go - but I’ll give this a try.) You might consider adding some of the key info from the first paragraphs in the docs (open source, self hosting) to the front page, above the fold. Github-link might imply it, but I was scanning for “open source” with my eyes and was initially disappointed not to find it and ready to dismiss the product right out of the gate.
- fallinditch 2y ago
  I didn't make this software, but heard about it on another thread. I was saying 'there should be an app like an 'auto-RAG' that scrapes RSS feeds and URLs' and be_erik wrote 'I do exactly this with hoarder. I passively build tagged knowledge bases with the archived pages and then feed it to a RAG setup.' What does your setup look like?
  - djhn 2y ago
    Things I bookmark (browsers), like and save (youtube, tiktok), upvote (hn) get archived to both internet archive and my own storage.

PeterStuer 2y ago

You can add '/display-feed.rss' to the list of common suffixes for many .eu sites

domysee 2y ago
Thank you - will do!

aucisson_masque 2y ago

I thank WordPress for most of my RSS feed. I follow mostly RSS on non technology website, for instance road cycling. people that wouldn't care or know about RSS because they are not very techy, yet because they are normies that use WordPress for all their website it puts a page with RSS feed automatically. You got to find it with developer tool by searching RSS but 99% of the time if it's WordPress it got RSS. Thank you WordPress you bloated piece of shit :)

openrisk 2y ago

Its interesting to contemplate an RSS-first browser that would have this functionality built-in. Think for example of promoting to full browser status a desktop RSS reader like Akregator [1] (which already embeds a webview). The browser as we now know it is mostly a static application that has long lost its user-centric mission. Websites might push some stuff but the user must do thinks manually. Its primary function is to provide a search window to external search. People even stopped using bookmarks and search for everything. This hypothetical RSS-Browser could become the main organizational tool for the users web experience, integrating the use of bookmarks. In fact even more "feeds" could be integrated like email and activitypub or atproto posts. It boils down to the fact that each person has a number of profiles/roles and within each they have a taxonomy of interests and we need a tool that integrates static and dynamic sources of information. [1] https://apps.kde.org/akregator/ https://apps.kde.org/akregator/

renegat0x0 2y ago

I fought this problem, since I wrote my own RSS reader in python. Might not be perfect. The problem with the approach presented here is speed. Most of the web pages, especially smaller are really slow. Crawling most of the web pages is pain, especially if you use selenium and small SBC. Therefore either the page presents a clean nice RSS link, or get lost. Most of the good, modern pages give you nice RSS. Even GitHub gives you RSS for commits. For other pages I try openRSS. For YouTube I use yt-dlp to obtain channel id, to establish RSS. Algorithm is crude, but gets the job done. https://github.com/rumca-js/Django-link-archive/blob/main/rsshistory/webtools/url.py https://github.com/rumca-js/Django-link-archive/blob/main/rs...

ewired 2y ago

Pasting a URL in NewsBlur also uses several of these techniques to find the feed(s), and it is open source, so the feed-finding code could be ripped out of NewsBlur as an alternative to this.