10 ms·
Show HN: RSS feeds for arbitrary websites using CSS selectors
- spicybright 5y agoI very simple plumbing tools like these.
- deleted 5y ago[deleted]
- toastal 5y agoThis reminds me of the Soupault (https://soupault.app/ https://soupault.app/) philosophy for building static sites. You write it in any language you want like, pass it trough Pandoc or AsciiDoctor as preprocessor, and postprocess with Lua and CSS selectors.
- vhodges 5y agoYou both might find https://stitcherd.vhodges.dev/ https://stitcherd.vhodges.dev/ interesting then.
- rcarmo 5y agoThis is nice. I’m actually doing much the same with Node-RED’s HTML parser, which also supports simple selectors.
- Vinnl 5y agoThanks. I initially used a regular HTML parser as well, but I quickly ran into sites that wouldn't render without JavaScript. I'm therefore now using a regular browser controlled by Playwright to fetch the websites.
- axiolite 5y agoCare to name any sites? I've always managed to find workarounds for everything I've wanted to follow. Most websites want to be indexed by search engines, and googlebot doesn't do javascript. So sometimes a forged user agent is all you need. Occasionally, finding the actual json file and parsing the info you need out of it does the job. etc.
- moehm 5y ago> googlebot doesn't do javascript But it does. Primary source: I experienced it on my own site. Secondary source: https://developers.google.com/search/docs/guides/fix-search-javascript https://developers.google.com/search/docs/guides/fix-search-...
- chrismorgan 5y agoThis changes from time to time, of course, but when last I investigated, around two years ago, consensus was that it mostly wouldn’t do JavaScript until you nudged it into doing so in some way that I forget, and that it was always slower to index/update if it needed to do JavaScript. (For my part, I disable JavaScript by default for various reasons, mostly performance, and it’s decidedly uncommon for a general-internet site to be completely broken by it. Sites that get posted on HN are disproportionately JS-dependent, especially if they’re new.)
- Vinnl 5y agoThe User Agent trick is a good one that I should've tried, but I just checked and it didn't work for this one. Parsing actual JSON wasn't really an option, as I wanted to be able to quickly and easily add RSS feeds. Possibly SEO is less a concern for the type of website I initially made this for, i.e. Dutch real estate agents. Most people find their listings through funda.nl rather than through search engines; I was just hoping to see them listed before they got posted there. Send me a message on Twitter or email me (hacker_news@ my domain) if you still want the URL of a failing website to play around with.
- bellyfullofbac 5y agoI wish there's a snippet of what the XML looks like, even better if it's "rendered"...
- Vinnl 5y agoThat's a good one, I'll see if I can add something to the site. Meanwhile you can see the generated examples here: - https://vincenttunru.gitlab.io/feeds/funfacts.xml https://vincenttunru.gitlab.io/feeds/funfacts.xml - https://vincenttunru.gitlab.io/feeds/wikivoyage.xml https://vincenttunru.gitlab.io/feeds/wikivoyage.xml And the combined feed: - https://vincenttunru.gitlab.io/feeds/all.xml https://vincenttunru.gitlab.io/feeds/all.xml Add those links to your feed reader to see rendered examples while I update the site. Edit: preview added to the website.
- moehm 5y agoFYI, the links inside your feeds don't work, because they are relative, not absolute.
- Vinnl 5y agoAh, you mean the ones inside the contents? That's a good one. I'm not sure if that's easily fixable, but I'll give it some thought. For those interested, I'll track it here: https://gitlab.com/vincenttunru/feed-me-up-scotty/-/issues/1 https://gitlab.com/vincenttunru/feed-me-up-scotty/-/issues/1
- chrismorgan 5y agoThe appropriate way is to use the xml:base attribute, as demonstrated in the example at https://datatracker.ietf.org/doc/html/rfc4287#page-4 https://datatracker.ietf.org/doc/html/rfc4287#page-4.
- Vinnl 5y agoThanks! I'll look at that.
- k1m 5y agoVery nice! I work on Feed Creator - https://createfeed.fivefilters.org https://createfeed.fivefilters.org - which is similar, although unlike yours doesn't use a headless browser, so selecting Javacript-inserted elements isn't supported.
- Vinnl 5y agoThat looks great! I ran into a couple of feed generators when I first needed this myself, but they were either paid or clumsy to use. That's when I thought, "alright, I can build this myself, and then I can even use CSS selectors, since I'm comfortable with those anyway". Of course, that is the point I should've thought about explicitly searching for one that supported that :) Ah well. What I do like about my current approach is that I have full control without having to run my own server, which is nice.
- axiolite 5y agoHere's a list of RSS feed generators: https://github.com/AboutRSS/ALL-about-RSS#webpagehtml https://github.com/AboutRSS/ALL-about-RSS#webpagehtml
- k1m 5y agoThe more the merrier! :) And always nice to see more in the RSS space. Feed Creator is in PHP and unlike yours doesn't produce a static file. The CSS selectors, URL and other parameters are all embedded in the URL, e.g. https://createfeed.fivefilters.org/extract.php?url=http%3A%2F%2Fjohnpilger.com%2Farticles&item=.entry&item_desc=.intro&item_date=.entry-date The biggest issue I've found is people struggling to work out which CSS selectors to use. I wrote a blog post not that long ago to help people use the browser's developer tools to do that. Might help too with anyone trying to use this (although perhaps most of the HN audience doesn't need help here): https://www.fivefilters.org/2021/how-to-turn-a-webpage-into-an-rss-feed-pt2/ https://www.fivefilters.org/2021/how-to-turn-a-webpage-into-...
- fran-penedo 5y agoSince everyone is pitching their own, I built https://github.com/fran-penedo/rssify https://github.com/fran-penedo/rssify, which started as a fork of https://github.com/h43z/rssify https://github.com/h43z/rssify. The basic functionality is similar to Vinnl's: give it a URL and some selectors and it builds the RSS feed. From this, I added a few things: templates (if you want to subscribe to individual projects within a webpage, like fanfics in ao3), transforms (when the data is not quite the text of the DOM element), a flask server you can use to add new URLs you have a template for and update the feeds, and a userscript to add the current URL using the server.
- 0des 5y agoWoah this is cool! What you did with the setup documentation and the bit about automation was a nice touch, I wish more projects had this attention to the detail. Very simple, useful and elegant. Thanks for sharing this with us, Vincent!
- colordrops 5y agoThis seems like it could be used to build a simple scraper as well.
- Vinnl 5y agoI'd recommend using the tools I used for this directly if you're looking to do this. Playwright in particular: https://playwright.dev https://playwright.dev
- chrismorgan 5y agoI’m glad to see that you’re using Atom rather than RSS, but you’re still calling it RSS. Can I convince you to stop calling it that? It’s factually inaccurate, and keeps RSS, the inferior format, more popular by virtue of mindshare. Just call them feeds, the proper generic term.
- nvr219 5y agoIf OP didn't use the word RSS and said "atom feeds" I wouldn't know what they're talking about, so I recommend keep calling it RSS.
- slightwinder 5y agoThe generic name for this is Newsfeed (or webfeed according to wikipedia, but never heard this in the wild). Would you understand it with those names?
- moehm 5y agoSame problem with SSL encryption. But with both RSS and SSL, if the user use a modern client, he shouldn't bother if its internally using RSS, Atom, SSL or TLS. I for one know most of my pals do know about RSS, but haven't heard the term 'Atom feed'.
- chrismorgan 5y ago“SSL” is finally dying as a term; the significant majority of what I see calls it TLS now. But the problem didn’t exist in the same way with SSL/TLS: SSL was actively killed off in favour of TLS so that regardless of what you call it, you’re actually dealing with TLS. But with feeds, RSS is still supported, and so the mindshare problem happens: people hear about RSS and so implement the inferior and problematic RSS rather than Atom, because they’ve never even heard of Atom and so don’t know that it’s what they should have implemented instead, because it’s more robust. Calling feeds RSS perpetuates RSS, which is undesirable. I say just advertise it as “feeds”, not “RSS feeds”. Atom is an implementation detail, just like RSS should be. (Well, RSS should be dead in general feeds, with only podcast feeds keeping it alive.)
- lorey 5y agoIn case anyone wants to detect the selectors automatically, here's a small python library I wrote that does it for you: https://github.com/lorey/mlscraper https://github.com/lorey/mlscraper
- clickok 5y agoA good idea and very cleanly implemented. I imagine that there's a ton of other possible applications that don't require much modification to the code. Thanks for sharing!
- lorey 5y agoThank you so much, happy to hear! There's a more versatile version coming soon :)
- kwerk 5y agoThis looks interesting! Will you be adding an open source license?
- lorey 5y agoWill do, just haven't decided yet :)
- ricardo81 5y agoNice, but it's essentially scraping? Scraping is brittle. Some design changes and the whole thing breaks. Maintenance nightmare.
- ricardo81 5y agoNot sure for the downvote reason. Like I say scraping is brittle. I've done hundreds of scraping projects in the past. A recent one was taking 50 shopping sites looking for what products they covered. Safe to say you look at it a month later and the XPaths/CSS selectors/whatever scraping method you used has changed. This is why APIs exist.
- Vinnl 5y agoIdeally sites would just publish RSS feeds themselves, but not all of them do — let alone an API…
- ricardo81 5y agoIndeed, but the scraping workaround is brittle. We could make rules for all sites on the web that don't have a feed or API but it's not all that manageable. It also begs the question of why they don't make their content that way, perhaps it was by choice. Especially if you use this kind of tool to syndicate things. Not quite sure of the novelty of this one given that people have scraped things for decades. There's probably a wiki endpoint for their example, maybe, maybe not. A lot of wiki's stuff is free to download and they have extensive endpoints for acquiring data.
- eyedontgetit 5y agoRe-stating the "brittleness" over and over doesn't add any weight to your argument. Regardless of whether or not the solution is brittle - it is still a viable solution for users who don't care or mind maintaining it. It sounds like you're not the target user and that's ok...
- Vinnl 5y ago
- deleted 5y ago[deleted]
- pacman2 5y agoThank you, will try it out. Two things I am using: Twitter to RSS: https://github.com/RSS-Bridge/rss-bridge https://github.com/RSS-Bridge/rss-bridge Arbitrary RSS feeds: https://feedity.com https://feedity.com
- nreece 5y agoThanks for using and mentioning Feedity! We've since rebranded to New Sloth: https://newsloth.com https://newsloth.com, which is now a simple integrated feed builder, reader and clusterer/deduplicator, specially aimed for knowledge workers with hundreds and thousands of feeds to monitor daily. Besides visual selector-based feed generation, our API can auto-magically detect relevant selectors (in most cases) as well.
- pacman2 5y agooh no. My free feedity feeds will keep working?
- canada_dry 5y agoBrowsing OP's repositories was this gem: https://gitlab.com/vincenttunru/flatuscode https://gitlab.com/vincenttunru/flatuscode A VSCode extension. That adds farts on every keypress. That's all.
- hallway_monitor 5y agoPretty great for April 1st... Or whenever you find an unlocked machine!
- canada_dry 5y agoWhen I saw it, I did do the quick calc based on the last update: worked out to around Christmas. OP might have just been bored over the holidays.
- Vinnl 5y agoHaha yeah, came up with the idea and went right ahead and implemented it. Probably should do a Show HN on April 1st though — just set a reminder for next year.
- Vinnl 5y agoIf you're back in the office, I can recommend enabling it for just one particular programming language on your coworker's VSCode.
- alanchen 5y agoRelated: https://github.com/DIYgod/RSSHub https://github.com/DIYgod/RSSHub This perhaps has more flexibility and can deal with almost any website.
- spacec0wb0y 5y agoWhat do I do to get this working? So far I've forked feeds, edited feeds.toml, checked it out as a branch gh-pages, pushed the branch up to github. I can see the page at https://<username>.github.io/feeds/ https://<username>.github.io/feeds/ but https://<username>.github.io/feeds/actions https://<username>.github.io/feeds/actions is just a 404.
- Vinnl 5y agoAh, those instructions are unclear — as far as I know, you first have to go to https://github.com/<username>/feeds/actions https://github.com/<username>/feeds/actions to enable Workflows for your repository. Then, your feeds should be published to https://<username>.github.io/feeds/<feedname>.xml https://<username>.github.io/feeds/<feedname>.xml. Does that work?
- spacec0wb0y 5y agonot exactly. I saw below the instruction to run `npx feed-me-up-scotty` so I did and it generated the public/ dir and feeds. ok I managed to access the actions route by a different URL to the readme. I copied the pages.yml workflow from your repo. Few minutes later I could see my feed. Very nice! I need to clean up my selectors now!
- ianberdin 5y agoMy friend is working on a quite simple feed generator from any website, social media: https://rss.app https://rss.app. Maybe helpful.
- pwrplus1 5y agoSome feedback for the project, please reveal pricing/limits before signup.
- goleary 5y agoAgreed, they should show it before signup. For anyone curious: Free - 2 RSS feeds 24 hour refresh rate $9.99 - 40 RSS feeds 30 min refresh rate $19.99 - 100 RSS feeds 15 min refresh rate My company uses RSS.app & I find it extremely nice to work with. You don't even have to provide selectors/click on elements for the vast majority of websites, they do that all for you. We could roll our own solution using any of the above offerings (or something built in house), but it's cheap enough for our usecase that we don't see a point.
- markdown 5y agoGreat project, but that's a shitty name.
- remram 5y agoCouldn't those selectors be maintained by the community? Instead of everyone deploying this on their own GitHub Actions, and having to fix it independently when it breaks, a single repo with all kinds of feeds maintained by everyone?
- Vinnl 5y agoThat's an interesting idea: something like DefinitelyTyped, but instead of type definitions for npm packages it provides selectors for URLs. Main challenge there would be organising the moderation, I suppose.
- contingencies 5y agoIt should be feasible to analyse the structure over time of the extracted data. Therefore, any proposed change which breaks the anticipated rhythm would be suspect.
- deleted 5y ago[deleted]
- toastal 5y agoWhy would a project on GitLab be running GitHub Actions? The project's doc acknowledges that git is distributed and can run an automation script of the user's choice of environment, and so should our language match this rather than implicitly endorsing a centralized service.
- judge2020 5y agoI think they simply meant 'instead of everyone running their own CI', not GH Actions in general - they're asking for a centralized feed repository and not one that is subject to people putting in work every few months to fix things that break, as that's likely to hurt adoption. In my opinion it sounds like a request for an open source/more community maintained feedly.com.
- 5y ago
- cyborgx7 5y agothis is something I've been looking for neat
- mhitza 5y agoThis could nicely supplement my GitHub automation that emails feed digests https://github.com/mhitza/subscriptions-digest https://github.com/mhitza/subscriptions-digest Similarly to my repository, I think I would suggest the option to fetch the configuration file from an external resource defined via an action secret. For my automation I'm using a Gist (not sure if Gitlab has same thing; also private but publicly accessible snippets). At least that way you can keep your own feed configuration while allowing those that fork the repository to not have to manually fix conflicts within the feeds.toml config.
- kdbg 5y agoKinda on a related note I found myself needing to make a bunch of these sorts of scraped feeds. The problem for me was the lack of date parsing support which I sorely needed (and it doesn't appear like this option supports it either) I ended up writing my own CLI tool that similarly supports CSS selectors for feed generation: https://github.com/dayzerosec/feedgen https://github.com/dayzerosec/feedgen I did write it specifically for my use-case so there are some "warts" on it like custom generators for HackerOne and Google's Monorail bug tracker. But perhaps someone else might benefit from its ability to create slightly more complicated RSS, Atom, or JSON feeds. Example config with date parsing: https://github.com/dayzerosec/feedgen/blob/main/configs/bishopfox.json https://github.com/dayzerosec/feedgen/blob/main/configs/bish...
- peanutz454 5y agoOn a job website I wondered if I could use CSS selectors to get to the important stuff. I think a lot of people have had those thoughts, and the website knew. While the page might have some repeatable structure, the ids and classes were just randomly generated. Also, After you open the job listing, they too were all different because the job poster had the ability to set up their own mini html that was embedded in the page.
- VoidWhisperer 5y agoIt is entirely possible that they did this to prevent scraping, but it is also possible it isn't intentional like that - for example, if you use the 'styled components' react library, it ends up spitting out a ton of class names that are basically gibberish, not intentionally.
- k1m 5y agoThat's been my experience too, as well as encountering a lot of legacy HTML markup without many useful class/id attributes. The random values are likely result of modern build tools like the one you mention, but there's also the growing popularity of utility-first CSS frameworks like Tailwind which also make finding useful selectors difficult. In these situations it might be best to rely on position of elements and other CSS selectors rather than attribute values. Unfortunately the :nth-child and :nth-of-type selectors still trip many people up. In Feed Creator we borrowed from XPath to make selecting by position a little easier. We've got a comparison here: https://help.fivefilters.org/feed-creator/css-selectors.html#element-by-position https://help.fivefilters.org/feed-creator/css-selectors.html...
- dugite-code 5y agoAlways good to see RSS projects pop up on hackernews. I'm still maintaining the Feediron plugin for TT-RSS - https://github.com/feediron/ttrss_plugin-feediron https://github.com/feediron/ttrss_plugin-feediron Unlike this project Feediron is only for modifying existing RSS feeds to extract the desired information. Typically uses xpaths to select content
- earthboundkid 5y agoI made a thing like this once to scrape school websites to figure out if there was a snow day. I found that most schools have Twitter and it was simpler and faster to subscribe to Twitter for updates.
- earthboundkid 5y agoThe project: https://github.com/baltimore-sun-data/track-changes https://github.com/baltimore-sun-data/track-changes
- DeLopSpot 5y agoIt seems that RSS feed generators are a bit like static site generators: it's often thought to be easier to make your own than to learn to use someone else's. Anyway, here's another self-hosted open source RSS feed generator for arbitrary websites: https://github.com/hueyy/HungryHippo https://github.com/hueyy/HungryHippo
- Vinnl 5y agoI mean, it's exactly like a static site generator — I'd call it JAMstack, except the "API" is a plain HTML page and the markup is RSS :) So yeah, definitely straightforward enough for a case of NIH syndrome. I think putting together the website took more time than writing the tool itself...
- lucideer 5y agoBecause the design of RSS/Atom put all of the complexity on the client (polling, state management, etc.) it's literally the same as static site generation. And by "the same", I don't mean "an equivalent but separate problem". I actually think having two separate generators—one outputting HTML, the other RSS—seems a bit wasteful. They're both parsing (presumably) the same content hierarchy and outputting it as SGML/XML-ish documents served over HTTP. One app should probably just do both (and it's easy to make your own that does)
- askhan 5y agoThis entire post has galvanized me to write up an idea I've been noodling over as I work on a reader myself: a standard that would eliminate precisely the waste you mention by specifying within the HTML all that's needed for a feed. See https://sfeed.org https://sfeed.org. In the spirit of the multi-meaninged RSS acronym itself, the S might stand for scrape, selector, speed, or of course Scotty. Vinni, might you be interested in enabling the standard in Feed Me Up?
- Vinnl 5y agoThere already is such a standard! https://microformats.org/wiki/hatom https://microformats.org/wiki/hatom Happy to take a merge request that adds the option to set `extends = "sfeed"` or `extends = "hatom"` and then automatically sets the correct selectors though. (That said, if a publisher goes through the trouble of adhering to those selectors, they might as well publish a feed while they're at it :)
- shadowbanned69 5y agohttps://weboas.is/tech/ https://weboas.is/tech/ quite a collection of tech RSS feeds
- flowerlad 5y agoI would love to have an RSS feed for stock quotes that works by scraping Google finance. A lot of people would love to have that as well.
- asdff 5y agoYou can do this with python libraries already
- tgel0 5y agoI created a free sheet-to-RSS service which lets you do that (somewhat). Here is a short write-up including the links: https://www.notion.so/How-to-create-your-own-stock-and-crypto-prices-RSS-feed-with-Sheets-and-crssnt-a6fded0d51244417aaa36439a7c107fe https://www.notion.so/How-to-create-your-own-stock-and-crypt...
- bjoli 5y agoWe were a bunch of friends having a "web ring" (remember those?) and I did something similar using XHTML and xpath back in 2001. We stopped when RSS became commonplace in 2003and predicted a wonderful future. Almost 20 years later someone reinvented it, but having to rely om styling rather any semantic or clear structural indormation. It makes me a bit sad. Not to piss on his/her parade, though: this is a great thing. I am going to tinker with it tonight.
- jazzdev 5y agoSo GitHub Actions is essentially a free server to run any background cron jobs I want?
- jazzdev 5y agoI guess not. Github terms[1]: Actions should not be used for: - any other activity unrelated to the production, testing, deployment, or publication of the software project associated with the repository where GitHub Actions are used. [1] https://docs.github.com/en/github/site-policy/github-terms-for-additional-products-and-features#actions https://docs.github.com/en/github/site-policy/github-terms-f...
- qwerty456127 5y agoThe problem with RSS/Atom feeds has always been they were mostly auto-generated. If only those was authored like a primary medium (instead of being just crappy auto-cuts of the articles' beginnings with links to the web site) it would be way better than social networks.
- galambo 5y agoRelated: https://galambo.wordpress.com/2021/04/21/rss-tools/ https://galambo.wordpress.com/2021/04/21/rss-tools/
- ossusermivami 5y agoI have a 7 years old litle girl and i have hard time explaining her CSS selectors, any chance someone can help me ? on the other hand she has no problem grasping the concept of XML and RSS feeds,