4 ms·
A lot of discussion around RSS revolves around the format for the data/metadata (e.g. the Atom feud) but the real problem with it is this: To consume an RSS fe
by PaulHoule 1y ago
A lot of discussion around RSS revolves around the format for the data/metadata (e.g. the Atom feud) but the real problem with it is this:
To consume an RSS feed you poll it. There are two polling speeds: too fast and too slow, and it's possible to be both at the same time.
Note the struggles of this Karen to turn RSS from a simple stateful protocol to a complex stated protocol, and she'll ban you if you ever reset your cache and rerun your fetcher because your cache got corrupted or you suspect it might have been corrupted.
http://rachelbythebay.com/w/2022/03/07/get/ http://rachelbythebay.com/w/2022/03/07/get/
You really want to have a stream of feed items and to be able to: (1) replay the whole stream all the way from or to the beginning and (2) query "what got added after time t?" and just get that. ActivityPub accomplishes this but people don't really like it. For Dave Winer it is all blub but even if he doesn't believe in the Fedi, he's on it.
I really like
https://superfeedr.com/ https://superfeedr.com/
because it does all the polling for you and hits your webhook whenever a new feed item appears. My webhook is about 15 lines of Python running as a Lambda function that posts items to an SQS queue and my YOShInOn RSS reader just drains the queue at its convenience. The pricing at 10 cents/feed/month is a bargain for high volume feeds like MDPI, arXiv, and The Guardian [1] but unfortunately I can't really afford to subscribe to 2000 little blogs that post maybe once a week at that rate. I wish there were more Planets.
https://en.wikipedia.org/wiki/Planet_(software) https://en.wikipedia.org/wiki/Planet_(software)
[1] AWS costs would be trivial in comparison even if it got out of the free tier
- jonchang 1y agoHer name is actually Rachel, not Karen.
- PaulHoule 1y agoI know. Perhaps I should have said "one of those misanthropic online personalities who kills 1000 contributions to open source you never knew you would have had because of their public bad attitude" and the fact that they have supporters is why Facebook wins and open systems lose.
- DamonHD 1y agoRachel is absolutely not like this in my experience. How rude!
- account42 1y agoThis is really not much of an issue if both sides implement HTTP caching (If-Modified-Since or Etags). Atom also adds pagination which allows you to keep all old items accessible to supporting readers while cutting down the main feed to just the last entry. The little bandwidth a well managed feed takes is really not worth giving up the ability to host the feed statically which the overly complex ActivityPub can't do.
- PaulHoule 1y agoYou are talking about this sort of thing? https://datatracker.ietf.org/doc/html/rfc5005 https://datatracker.ietf.org/doc/html/rfc5005 Http might be better in 2025 than it used to be but historically the cache is as much a problem as it is a solution. That is, when you have heisenbugs the answers are frequently "clear the cache" [1] and "don't use the cache" [2] and it's often a better strategy to "rip and archive the whole site right now and come back in six months and make a fresh rip" if you're working on a large scale and can tolerate being up to six months out of date. In general database-backed sites struggle to implement if-modified-since since a fully correct implementation has to traverse the graph of all database objects that are looked up in the request which costs about as much as… the request. Your cache needs a cache and you can make it work at the system level if you have a modified date cache and always invalidate it properly. If you are doing that you might as well materialize a static web site fronting your site the way the Wordpress supercache works —- then you’ll find your site crashing a lot less! I’ll admit ActivityPub is complex but http caching is complex in a particularly poisonous way in that there are so many alternate paths. An ActivityPub system (client+server) could be 100% correct but an http-based system might not have a clear line around it and might never be quite right. A stateless system could run for 10 years without trouble, it might take you 10 years to realize a stateful system was feeding you corrupted data. [1] fix it... for now [2] ... and they're no longer part of your life
- account42 1y ago> In general database-backed sites struggle to implement if-modified-since since a fully correct implementation has to traverse the graph of all database objects that are looked up in the request which costs about as much as… the request. That's only true if your implementation favors editing performance over GET request performance ... in which case you get what you ordered. If your feed requires any database connections then that's a choice you made. I disagree that HTTP caching is complex. The basic primitive are: - An indication until when the returned document can be used without checking for updates. - A way to check if a cached document is still valid (based on either modification time or id). This provides everything you need to implement an efficient feed reader.
- Karrot_Kream 1y agoDo you know (or have an idea about public discussions indicating) why ActivityPub never ended up being used for feeds? AP seems to mostly be Mastodon and Lemmy but it obviously has a bunch of fixes for problems inherent in RSS, Atom, and OPML.
- PaulHoule 1y agoI’ve seen a lot of Dave Winer saying his blub is good enough and abstractly that ActivityPub is too complex. There’s pixelfed too. Practically if I want to have an ActivityPub feed for my blog I might host in on Mastodon and if the character limit is too little you can have a server like https://writefreely.org/ https://writefreely.org/ If anything characterizes the RSS community it is a lack of imagination and rejection of the last 25 years of work in relevance, ML and UX. RSS readers are still failing with the same failing interfaces that failed 25 years ago. That said, the people who are still using it feel satisfied and other than Rachel that aren’t a lot of people who will try to stop you if you do try something ambitious. You just gotta poll poll poll and poll poll poll some more or pay somebody to poll poll poll for you.
- tolerance 1y agoIt’s a pity that this is the bottom-most comment and equally it’s a shame that the slur “Karen” made its way into the White lexicon spoiling an otherwise informative remark. Correct me if I’m wrong, but is Winer’s somewhat recent effort with FeedLandⁱ any different from Planet? ⁱ: https://feedland.org/?username=scripting https://feedland.org/?username=scripting
- PaulHoule 1y ago(1) Some people read from the bottom up (2) I really am mad at Rachel for this and that's from someone who's been writing webcrawlers [1] since 1998, been an RSS innovator, and been responsible for complex systems when they fail. (3) Maybe I am missing it but I don't see an actual feed for that URL, I see only an OPML file. Dave is really gay [2] for OPML files but I'm not because I still have to work to fetch all the items. Yet, visually the OPML file and blogroll look like a planet and you're not the first person who's pointed Dave's blogroll as a solution as opposed to the problem that I see it is. (4) Looking at the head of the list I think "Daily Kos" and "404 Media" suck but that I already subscribe to many of them like "Arstechica" -- looking at the tail of the list I see there are gems that I'm not getting. If those things were getting aggregated topically it would please both me and Rachel. [1] that don't crash your server [2] in a good sense!
- tolerance 1y ago> If those things were getting aggregated topically it would please both me and Rachel. I think I catch what you’re getting at and it reminds me of what seems to be a primary gripe with RSS—compromising an otherwise decent spec as far as I can tell—discoverability and curation. I can’t grasp the more technical issues surrounding polling just yet so I can only intuitively get how it corresponds with the other issues. In spite of this, and leaning again on my intuition, I didn’t get a good feeling from Rachel’s blog post when I first came across it either. And yeah I’m not sure how FeedLand works under the hood, I just matched Wikipedia's description of a planet with what I knew FeedLand looks like on the frontend. I also know there’s been some “folksonomic" efforts for curating lesser-know feeds, but to be honest I’m not fond of what these networks of folks appear to be into. Compare your distaste for Daily Kos and 404 Media (which I share) to the vibe I’m trying to avoid on that front.