4 ms·
Feedmaker: URL + CSS selectors = RSS feed
- mustaphah 1y agoThe good news: made it to the front page. The bad news: so did the 503 page.
- benbristow 1y agoIn some ways a good thing, no? Shows you've got work to do on optimisation for large audiences. A free stress test (unless you're on a host that charges per hit or bandwidth excess), as you will. Did load eventually for me, thought it was broken as no styles but looks like it's intentional.
- uyzstvqs 1y agoSeems to be hosted using fly.io
- deleted 1y ago[deleted]
- kschaul 1y agoGlad you’re find the tool interesting! A short blog post behind it: https://kschaul.com/post/2023/04/16/feedmaker-quickly-generate-an-rss-feed-from-any-website/ https://kschaul.com/post/2023/04/16/feedmaker-quickly-genera... And the GitHub url (hopefully easy to host your own instance): https://github.com/kevinschaul/feedmaker https://github.com/kevinschaul/feedmaker
- mustaphah 1y agoLooks like you're hosting this on fly.io - PAYG model. You could probably host this for free on Cloudflare Workers; 100k requests/day on the free tier; static content (the homepage) is free & unlimited. Edit: The catch is the 10ms CPU cap per request - you'd need a super lean implementation. Django's too heavy for that.
- 0cf8612b2e1e 1y agoPython alone is many milliseconds to start. Unless they give you some allowances for interpreter overhead.
- mustaphah 1y agoWell, someone already did with JS: https://github.com/ProfessorManhattan/rss-worker https://github.com/ProfessorManhattan/rss-worker
- bradbeattie 1y agohttps://github.com/RSS-Bridge/rss-bridge https://github.com/RSS-Bridge/rss-bridge is what I've been using for the same purpose.
- zekenie 1y agoNot the same but this gives me an idea… what if there was a map reduce for doms as a web primitive. Like imagine if I could make a dom (or feed) that was some selection and transformation of another dom
- onedognight 1y agoYou have just re-invented XLST.
- 1-more 1y agohttps://www.w3schools.com/xml/tryxslt.asp?xmlfile=cdcatalog&xsltfile=cdcatalog https://www.w3schools.com/xml/tryxslt.asp?xmlfile=cdcatalog&... give it a whirl!
- jeroenhd 1y agoIn the days of XHTML, XSLT would be exactly what. The modern web equivalent would require XSLT but for HTML/SGML (SGSLT?).
- int0x29 1y agoI made a CGI program that ran CSS selectors against URLs and returned the output. I debated making it public and then realized I probably didn't want to run an open proxy. I'm curious how long this will last.
- oneeyedpigeon 1y agoYou could just have made it public and self-hosted.
- crazygringo 1y agoI love this. Has anyone tested to see if it works with Blogtrottr which will email you whenever there's a new item in an RSS feed? Just since this doesn't seem like it even includes a date field in the RSS? And of course no guid. So I'm wondering how compatible it winds up being.
- kevincox 1y agoDates shouldn't matter. The feed has ID elements which is what identify entries. Atom has no guid element. So I would expect this to work with any reader.
- edoceo 1y agoI wish they had concrete, accurate id and created_at. IIRC these attributes are fixed in AT.
- crazygringo 1y agoBut is this producing ID elements? And if so, based on what, since they don't seem to be coming from any CSS selectors? That's my question.
- kevincox 1y agoIt seems to use the link as the ID based on clicking a few examples on the site. An ok option for this type of thing.
- ZYbCRq22HbJ2y7 1y agoShould be able to achieve this without selectors with HTML to Markdownish (something like Firefox's Reader mode).
- cleartext412 1y agoOh, so this is what Reader mode does. The few times I actually tried it, it worked badly, with huge chunks of text content missing from the page. Makes me wonder if with modern web the task has became so difficult even a browser couldn't pull it off, or if they just wasn't trying to do a good job with the feature.
- mg 1y agoThat is a good idea. 59 requirements, including Django, seems pretty heavy though? For my own RSS feed, I use this 48 line Python file with no dependencies outside the standard library: https://github.com/no-gravity/atomfeed.py https://github.com/no-gravity/atomfeed.py It takes an array with the entries as input, not a web page. But I guess the HTML parsing should take no more than another few lines? For HTML parsing, I have good experiences with the lxml module which is in the Debian repos. It is fast and works pretty well.
- oxalorg 1y agoI recently added the python-feedgen module for creating feeds in my blog generator: https://github.com/oxalorg/genox/commit/3a73013ffe82930b1a7ea587f159c58f624a4b0c https://github.com/oxalorg/genox/commit/3a73013ffe82930b1a7e... I always love removing dependencies and simplifying software. I will try and switch to a simpler implementation like yours, thanks for sharing!
- ulrischa 1y agoSame can be done wirh freshrss
- msephton 1y agoIf you're serious about RSS curation and reading, FreshRSS really is the Swiss Army Knife. It does so much, including this“any site/page to RSS”. My favorite feature is that it makes refreshes in your feed reader client pretty much instant, which is such a huge quality of life improvement.
- gottlobflegel 1y agoYou can just use an XSLT stylesheet like this: https://wwwcip.cs.fau.de/~oc45ujef/misc/src/atom.xsl https://wwwcip.cs.fau.de/~oc45ujef/misc/src/atom.xsl xsltproc includes a handy --html flag that lets you just process the source file directly.
- deleted 1y ago[deleted]
- pkal 1y agoCan you also generate+use the XSLT stylesheet dynamically from a form input so that you can use a single meta-stylesheet for multiple sites? Oh, and is you brother coming to the party?
- danielheath 1y agoI wrote a similar thing in go (using chromedriver, so it could handle things that need JS). Handled most things nicely, but I found a few sites where I wanted multiple selections to be combined into one document. I emailed the result to myself, turning any images into attachments; this meant my “feed reader” had read/unread tracking that synced across devices, some html support, folders, offline viewing, etc.