5 ms·
RSS-proxy: create an RSS/ATOM or JSON feed of almost any website
- 1vuio0pswjnm7 5y ago"... almost any website." What are the domain names of some (dynamic) websites where this will not work.
- marginalia_nu 5y agoOnly guessing, but https://oeis.org/ https://oeis.org/ ?
- lostsoul8282 5y agoThis is great. Many scrapers require constantly monitoring which is a bit heavy for the scraping process. I can see this tool making it easy to monitor for changes fairly quickly. Great work!
- smbv 5y agoAs a big fan of RSS (I read HN through hnrss[0]), this is wonderful. It seems to be a nice catch-all for whatever websites rss-bridge doesn't support. If any of you are looking into RSS, check out FreshRSS[2] (an RSS aggregator). A mobile client I especially like is NetNewsWire[3] (for Apple products only, but is imo amazing) [0] https://hnrss.github.io/ https://hnrss.github.io/ [1] https://github.com/RSS-Bridge/rss-bridge https://github.com/RSS-Bridge/rss-bridge [2] https://www.freshrss.org/ https://www.freshrss.org/ [3] https://netnewswire.com/ https://netnewswire.com/
- rambambram 5y agoAh, I like FreshRSS, based on what I can see from it. It's written in PHP and free.
- stanislavb 5y agoThanks! RSSHub is another option we could add to the mix https://github.com/DIYgod/RSSHub https://github.com/DIYgod/RSSHub. Also, all projects mentioned on this thread can be tracked and easily compared on LibHunt https://www.libhunt.com/posts/500676-rss-proxy-create-an-rss-atom-or-json-feed-of-almost-any-website https://www.libhunt.com/posts/500676-rss-proxy-create-an-rss...
- mderazon 5y agoNetnewawire looks great! Any similar recommendations for Android ?
- phil294 5y agoTo elaborate on this, RSS-Bridge maintains custom extraction rules for many sites [1], but RSS-proxy attempts to do that for any site using some pretty nifty logic [2]. I tried it on a few pages and it seems to do its job accurately, if the HTML is good enough. Semantic HTML5 can be so helpful, especially in regards to tools like this one, or crawlers, or generally accessibility-wise. Unfortunately, its syntax is not enforced by browsers (for good reasons), so many developers never leverage it - and we end up with many custom-built sites falling back to <div>s everywhere, JS click handlers on non-interactive elements etc. I really wish modern framework docs and guides would more strongly hint to accessibility concerns. [1] https://github.com/RSS-Bridge/rss-bridge/tree/master/bridges https://github.com/RSS-Bridge/rss-bridge/tree/master/bridges [2] https://github.com/damoeb/rss-proxy/blob/master/packages/core/src/feed-parser.ts#L407 https://github.com/damoeb/rss-proxy/blob/master/packages/cor...
- lelandfe 5y agoSeconding NetNewsWire. A recent version added iCloud sync so your feeds and unread statuses sync across the desktop and mobile version. Somehow it's free.
- thirdplace_ 5y agoGood work there. Imagine if we had a universal feed generator that could generate a feed out of everything. No more manual checking labor.
- smbv 5y agoRSS saved me the "right click on a bookmark folder -> open all" every hour or so. Does RSS increase my productivity or decrease it? On one hand, I'm not obsessively checking a bunch of websites. On the other hand, I have a firehose of content directed at my face with hundreds of feeds.
- walterbell 5y agoNext step is local analytics to sort articles based on local-only rules/analytics/goals.
- acidburnNSA 5y agoRSS Hub also attempts to be exactly that: https://docs.rsshub.app/en/ https://docs.rsshub.app/en/
- mmahemoff 5y agoInteresting idea, but the fact so many of the feeds are broken probably indicates how difficult this is to maintain. Ideally something that needs to be done via algos and machine learning rather than custom parsing for each site (not unlike what search engines already do internally).
- busymom0 5y agoThis is quite useful, especially the self host option. What type of websites don't work?
- shellmachine 5y agoGreat tool, something I didn't know I need in my life. Thanks.
- eric4smith 5y agoA good way to find new pages on a site is to use the site map feed in xml or text format. Then look for open graph tags in the page for titles descriptions, canonical urls and images. Most modern websites have all of those these days and they are automatically updates. Making an RSS feed reader is thus easier and less prone to errors from analyzing the markup of pages. Neat product though.
- xrd 5y agoGreat idea, but it seems it does not work for basic static sites like those generated by SvelteKit (http://svekyll.com http://svekyll.com generated this site: https://webiphany.com https://webiphany.com). That's too bad.
- pedro1976 5y agoAuthor here, thanks for those tough nuts :) I am working on a successor of this POC called rich-rss [0], which contains a full rewrite of rss-proxy in a strongly typed language. [0] https://github.com/damoeb/rich-rss https://github.com/damoeb/rich-rss
- lapinot 5y agoWell that site is a bunch of divs. Afaik the tool leverages semantic html (section, article etc).
- thedougd 5y agoA replacement for Yahoo Pipes! In the early Android days, I wrote a few apps that relied on web site scraping to provide their data. Even back then, these websites evaded scraping. Yahoo Pipes to the rescue. You'd set it up to scrape their site and provide a well-formed RSS feed of the data. Since these sites saw Yahoo connecting, they typically didn't block it, presumably because they thought it was search crawling.
- ss2f 5y agoMonitoring Public Facebook pages? A very painful process. Facebook requires a login to view a public pages -- and most public offices like governments post content exclusively on Facebook so as things stand, you have to have an account on Facebook to access government communication, which is plain sad. I do not remember any good tool to access Facebook pages. If you get one (rss-hub), it breaks in a week after Facebook changes their API endpoints. If you have one, let me know.
- dirtyid 5y agoRSS scrapers I want: something that consolidates twitter threads like threadreaderapp into articles.
- davidpfarrell 5y agoThrowing https://heartfeedrss.wordpress.com/usage/ https://heartfeedrss.wordpress.com/usage/ into the ring as its been my go-to rss reader since LinkedIn's Pulse was decommissioned. I really dig the grid-based approach for both increased content density and easy cross-feed navigation.
- smusamashah 5y agoIt doesn't work well everywhere and there isn't a way to customize its selection of divs e.g. see this thread itself https://rssproxy-v1.migor.org/?url=https:%2F%2Fnews.ycombinator.com%2Fitem%3Fid%3D29369706 https://rssproxy-v1.migor.org/?url=https:%2F%2Fnews.ycombina...