4 ms·

Question about the RSS spec: When pulling RSS, how do you know how often to poll? How do you know which items have been seen previously?

oneeyedpigeon 2y ago
> how do you know how often to poll? I would guess a combination of frequency from sitemap.xml, last modified http header, and past heuristics. Previously viewed items would, I think, need to be cached in the client (unless the RSS URL uses some kind of token to identify the user, which sounds ripe for abuse).
octochamp 2y ago
RSS can be polled as frequently or infrequently as you like, it's just a bit of XML hosted by a site which lists content or links to content. Tracking what's been read or not isn't done by the RSS feed or whoever hosts it, it's performed by the user's feed reader, which be just a local app on your phone or PC, or it might be a cloud service, either hosted (like Feedly) or self-hosted (using ie. FreshRSS).
6510 2y ago
You could examine the time between feed items. Personally I just start my reader then it aggregates and sorts my feeds by date into a single interface. This works well specially for much larger numbers.
asjo 2y ago
rachelbythebay has a service and a series of blog posts about the technical side of this, starting at https://rachelbythebay.com/w/2024/05/27/feed/ https://rachelbythebay.com/w/2024/05/27/feed/ TL;DR: readers should not poll more often than once and hour, use ETag and If-Modified-Since to determine whether to download the full feed again. Which items you have seen previously is something the feed reader keeps track of.
- fire_lake 2y ago
  > Which items you have seen previously is something the feed reader keeps track of. Is there a particular field that can be used as an identifier?
  - asjo 2y ago
    Yes, the `guid` field in RSS and the `id` field in Atom.
    - account42 2y ago
      A lot of readers (intentionally?) get this wrong though and show entries as new/unread if there are changes to the content. Of course a lot of feeds also get this wrong and change the GUIDs for existing entries once in a while which results in strictly compliant readers showing you the entire feed history as new. Really annoying.
      - asjo 2y ago
        My system updates¹ the entry and marks the changes in the first case. What is right and what is wrong depends on what you want, I think. The latter is annoying, I agree. ¹ It is an NNTP interface so the article is superseded; https://feedbase.org/about/ https://feedbase.org/about/ - if you don't want to see updates, you can configure your newsreader to skip supersedes.
- account42 2y ago
  Polling every hour is excessive and will get you temp-blocked on some sites. It really depends on you but IMO for most feeds polling once a day is plenty.
domysee 2y ago
RSS actually has fields for it, but I've never seen it used. https://www.rssboard.org/rss-specification#optionalChannelElements https://www.rssboard.org/rss-specification#optionalChannelEl... ttl, skipHours, skipDays Source: working on a feed reader (https://lighthouseapp.io/ https://lighthouseapp.io/) and feed finder tool (https://lighthouseapp.io/tools/feed-finder https://lighthouseapp.io/tools/feed-finder).

oneeyedpigeon 2y ago

> how do you know how often to poll? I would guess a combination of frequency from sitemap.xml, last modified http header, and past heuristics. Previously viewed items would, I think, need to be cached in the client (unless the RSS URL uses some kind of token to identify the user, which sounds ripe for abuse).

octochamp 2y ago

RSS can be polled as frequently or infrequently as you like, it's just a bit of XML hosted by a site which lists content or links to content. Tracking what's been read or not isn't done by the RSS feed or whoever hosts it, it's performed by the user's feed reader, which be just a local app on your phone or PC, or it might be a cloud service, either hosted (like Feedly) or self-hosted (using ie. FreshRSS).

6510 2y ago

You could examine the time between feed items. Personally I just start my reader then it aggregates and sorts my feeds by date into a single interface. This works well specially for much larger numbers.

asjo 2y ago

rachelbythebay has a service and a series of blog posts about the technical side of this, starting at https://rachelbythebay.com/w/2024/05/27/feed/ https://rachelbythebay.com/w/2024/05/27/feed/ TL;DR: readers should not poll more often than once and hour, use ETag and If-Modified-Since to determine whether to download the full feed again. Which items you have seen previously is something the feed reader keeps track of.

fire_lake 2y ago
> Which items you have seen previously is something the feed reader keeps track of. Is there a particular field that can be used as an identifier?
- asjo 2y ago
  Yes, the `guid` field in RSS and the `id` field in Atom.
  - account42 2y ago
    A lot of readers (intentionally?) get this wrong though and show entries as new/unread if there are changes to the content. Of course a lot of feeds also get this wrong and change the GUIDs for existing entries once in a while which results in strictly compliant readers showing you the entire feed history as new. Really annoying.
    - asjo 2y ago
      My system updates¹ the entry and marks the changes in the first case. What is right and what is wrong depends on what you want, I think. The latter is annoying, I agree. ¹ It is an NNTP interface so the article is superseded; https://feedbase.org/about/ https://feedbase.org/about/ - if you don't want to see updates, you can configure your newsreader to skip supersedes.
account42 2y ago
Polling every hour is excessive and will get you temp-blocked on some sites. It really depends on you but IMO for most feeds polling once a day is plenty.

domysee 2y ago

RSS actually has fields for it, but I've never seen it used. https://www.rssboard.org/rss-specification#optionalChannelElements https://www.rssboard.org/rss-specification#optionalChannelEl... ttl, skipHours, skipDays Source: working on a feed reader (https://lighthouseapp.io/ https://lighthouseapp.io/) and feed finder tool (https://lighthouseapp.io/tools/feed-finder https://lighthouseapp.io/tools/feed-finder).