4 ms·

A feed is not a block chain; the feed publishers are free to do anything on their feeds. So the feed readers must be able to protect themselves from serving del

A feed is not a block chain; the feed publishers are free to do anything on their feeds. So the feed readers must be able to protect themselves from serving deleted items in perpetuity. I ran a server-side feed collector in roastidio.us for airss.roastidio.us users, my algorithm is very simple: I keep only the top N items in the latest version of the feed; everything else are dropped. For a fast moving feed, if the users don't read soon enough some items will be missed. This is still better (in terms of usability) than the bombardment of items to a user just coming back from a long vacation.

kevincox 5y ago
The problem is that RSS doesn't really have a concept of "deleted". In general feeds just contain the newest items. Items not on the feed are generally considered old, not deleted. You can play with heuristics like "This feed is usually sorted by published date and these entries I saw previously should still be here but are missing, maybe they were deleted" but not even have published or updated dates and they aren't always accurate. This gets even more complicated with WebSub which allows hubs go remove entries they have already seen. So now you need to not do deletion on those requests, and if you turn down your poll rate for feeds with WebSub (which Feedly does IIRC) you might not even fetch the full feed to notice that items are missing for weeks. The one thing that would actually be reliable here is probably Atom's pagnation and archiving extensions which you can follow to get a view of all of the items in the feed and prune missing ones. However you probably don't want to do this every fetch (otherwise it kinda defeats the point of pagnation). Maybe you could combine this with other heuristics to come up with a decent system. But I'm not even sure these have ordering guarantees so I'm not sure if you are going to fetch potentially unlimited pages just to check if an item was deleted. If we did want to solve this problem the best option would probably be to add a tombstone element to the feed to explicitly remove items. TL;DR as far as I am aware neither RSS or Atom have the concept of deletion. Much like sending someone 100 email newsletters unrelated to their subscription will piss them off with no recourse try not to send them 100 feed updates that they don't want. Edit: It looks like there are tombstone proposals for both RSS and Atom: https://oleb.net/blog/2015/11/rss-feed-item-deletions/ https://oleb.net/blog/2015/11/rss-feed-item-deletions/. Maybe the author could have tried this, but IDK if anyone supports them.
kevincox 5y ago
Different conversion for a different topic. > For a fast moving feed, if the users don't read soon enough some items will be missed. This may work well for you but definitely isn't what I prefer. For example I often subscribe to recording feeds for conferences or other events. These feeds don't post for most of the year but them might have hundreds of entries. I definitely don't want my feed reader to drop all but the last 10 (YouTube for example keeps much less than the latest 100 videos in the feed). I want it to keep them all so that I can work through the list and watch the ones I am interested in. So while your some solution might work in some cases this definitely isn't what I would want personally and in my opinion what Feedly did here was reasonable as a general behavior, even if it isn't optimal for this particular case. I'd rather have it handle the happy common case we'll then to hurt it for the rare case when someone makes a mistake, especially when the downsides for the mistake aren't that high.
- derekzhouzhen 5y ago
  > I want it to keep them all so that I can work through the list and watch the ones I am interested in. Your usage case can be better served by visiting the website directly, after you learn that there are some updates from the feed. There is no guaranty that a feed aggregator would not miss anything anyway; this is the the nature of the poll based delivery.
  - kevincox 5y ago
    Thanks for explaining my use case to me. - How to I remember where I am when I have watched half of the videos? - What if two events happen around the same time. Now I need to check both instead of them being in the same place. It turns out that leaving items in my feed reader and dismissing them as I watch or decide not to watch works incredibly well.
    - derekzhouzhen 5y ago
      You are using the feed to overcome the usability problem of a poorly designed website. I have seen my fair share of poorly designed websites so I can't argue with that :) My feed reader's intended use case is to following a large number of slow moving feeds like personal blogs, which are usually not bursty and not super important if a few items fell in the crack.
AnonC 5y ago
> I keep only the top N items in the latest version of the feed; everything else are dropped. For a fast moving feed, if the users don't read soon enough some items will be missed. I completely dislike this behavior. I saw this on Ars Technica some years ago, where it was said that only the most recent 20 items (?) would be in the feed. Ars can be a fast changing site, and since full text feeds are a paid privilege, I thought it was a disservice to paying users not to have the ability not to miss out on something just because they were busy with something else for a day or two.
- derekzhouzhen 5y ago
  I don't have the time to read everything. I'd rather miss something from the busy news site than miss something from independent bloggers.

kevincox 5y ago

The problem is that RSS doesn't really have a concept of "deleted". In general feeds just contain the newest items. Items not on the feed are generally considered old, not deleted. You can play with heuristics like "This feed is usually sorted by published date and these entries I saw previously should still be here but are missing, maybe they were deleted" but not even have published or updated dates and they aren't always accurate. This gets even more complicated with WebSub which allows hubs go remove entries they have already seen. So now you need to not do deletion on those requests, and if you turn down your poll rate for feeds with WebSub (which Feedly does IIRC) you might not even fetch the full feed to notice that items are missing for weeks. The one thing that would actually be reliable here is probably Atom's pagnation and archiving extensions which you can follow to get a view of all of the items in the feed and prune missing ones. However you probably don't want to do this every fetch (otherwise it kinda defeats the point of pagnation). Maybe you could combine this with other heuristics to come up with a decent system. But I'm not even sure these have ordering guarantees so I'm not sure if you are going to fetch potentially unlimited pages just to check if an item was deleted. If we did want to solve this problem the best option would probably be to add a tombstone element to the feed to explicitly remove items. TL;DR as far as I am aware neither RSS or Atom have the concept of deletion. Much like sending someone 100 email newsletters unrelated to their subscription will piss them off with no recourse try not to send them 100 feed updates that they don't want. Edit: It looks like there are tombstone proposals for both RSS and Atom: https://oleb.net/blog/2015/11/rss-feed-item-deletions/ https://oleb.net/blog/2015/11/rss-feed-item-deletions/. Maybe the author could have tried this, but IDK if anyone supports them.

kevincox 5y ago

Different conversion for a different topic. > For a fast moving feed, if the users don't read soon enough some items will be missed. This may work well for you but definitely isn't what I prefer. For example I often subscribe to recording feeds for conferences or other events. These feeds don't post for most of the year but them might have hundreds of entries. I definitely don't want my feed reader to drop all but the last 10 (YouTube for example keeps much less than the latest 100 videos in the feed). I want it to keep them all so that I can work through the list and watch the ones I am interested in. So while your some solution might work in some cases this definitely isn't what I would want personally and in my opinion what Feedly did here was reasonable as a general behavior, even if it isn't optimal for this particular case. I'd rather have it handle the happy common case we'll then to hurt it for the rare case when someone makes a mistake, especially when the downsides for the mistake aren't that high.

derekzhouzhen 5y ago
> I want it to keep them all so that I can work through the list and watch the ones I am interested in. Your usage case can be better served by visiting the website directly, after you learn that there are some updates from the feed. There is no guaranty that a feed aggregator would not miss anything anyway; this is the the nature of the poll based delivery.
- kevincox 5y ago
  Thanks for explaining my use case to me. - How to I remember where I am when I have watched half of the videos? - What if two events happen around the same time. Now I need to check both instead of them being in the same place. It turns out that leaving items in my feed reader and dismissing them as I watch or decide not to watch works incredibly well.
  - derekzhouzhen 5y ago
    You are using the feed to overcome the usability problem of a poorly designed website. I have seen my fair share of poorly designed websites so I can't argue with that :) My feed reader's intended use case is to following a large number of slow moving feeds like personal blogs, which are usually not bursty and not super important if a few items fell in the crack.

AnonC 5y ago

> I keep only the top N items in the latest version of the feed; everything else are dropped. For a fast moving feed, if the users don't read soon enough some items will be missed. I completely dislike this behavior. I saw this on Ars Technica some years ago, where it was said that only the most recent 20 items (?) would be in the feed. Ars can be a fast changing site, and since full text feeds are a paid privilege, I thought it was a disservice to paying users not to have the ability not to miss out on something just because they were busy with something else for a day or two.

derekzhouzhen 5y ago
I don't have the time to read everything. I'd rather miss something from the busy news site than miss something from independent bloggers.