4 ms·
Ask HN: How do RSS readers handle items missing pubDates?
I'm writing an RSS reader, and I noticed that items often miss publication dates. Regardless, other readers seem to be able to gather them anyway, I'm guessing through the actual site.
Is this generally done through web scraping, archival information, web crawler data, or something else entirely?
I'm asking because I would hate to resort to those types of methods as they are generally heavy on resources and unstable.
- deleted 4y ago[deleted]
- Minor49er 4y agoIt's possible that the feed reader attaches it itself based on when it appears in the feed since the reader will keep polling the same feed over time
- pipeline_peak 4y agoI’m not understanding your suggestion. It’s using Microsoft Syndication Feed to parse the RSS files which themselves are missing pubdate info. There’s nothing stored on disk. Feed I'm parsing http://rss.cnn.com/rss/cnn_topstories.rss http://rss.cnn.com/rss/cnn_topstories.rss
- 6510 4y agoIn this case most items have <pubDate>'s, for the ones without you use the feeds <pubDate> in the <channel>
- pipeline_peak 4y agoUsing the same channel pubdate for all the items with missing ones wouldn't be accurate. I'm using The Old Reader as an example and all of their missing items have different dates.
- 6510 4y agojust dates or date and time?
- pipeline_peak 4y agodate and time
- 6510 4y agoThey are probably polling the feed.
- pipeline_peak 4y agoWhat do you mean by this, they are listening for changes in the webpage?
- 6510 4y agoThey check the feed every x min, if a new item appears they use the current time (or the channel pupDate) as the item pubDate. There might be more accurate ways but this is good enough. How is your thing suppose to work? Are you going to load all the feeds for every visitor? Even if it is just for personal use, how many feeds are we talking about? If you pull 20 per second and have only 200 the 10 seconds delay is going to be more annoying than it needs to be. You should probably consider storing the feeds on disk or in a db.
- pipeline_peak 4y agoMy goal is to load all feeds for every visitor, 200 feeds at first but ideally more. Like another person suggested, I will likely use Internet archive to make an api request and retrieve date time. If that’s inaccurate then I will use polling, but that does seem expensive. Are you suggesting storing the actual rss xml file in a database to be reused efficiently?
- 6510 4y agoThere always is a date some place. Find or derive them, order them by how resource heavy and/or crappy they are. Chose wisely for the type of reader you are making and/or create configuration options for it.
- pipeline_peak 4y agoUnfortunately, I think that's what I'm going to have to do.
- 6510 4y agono no, this is the fun of building the rss/atom reader! My funniest moment was when I didn't notice someone deleted their feed and the missing page redirected to the front page. The many layers of failbacks just parsed the html as if it was a feed. It didn't have <item>s so it tried to parse it without items, it didn't have <link>s or <guide>s so it ended up looking for <a>'s in the <description> but it didn't have <description> so it just accepted any <a>'s and used whatever was in the <a>'s as the <title>.
- pipeline_peak 4y agoI still think it will be fun in an inconsistent, man made jungle of the web sort of way. And I agree the types of mishaps are really interesting because they almost can't be replicated since they depend on so many environmental factors they almost appear physical. There's an interview somewhere on Youtube of Bjarne Stroustroup, I think it was one of those Big Think videos. He was asked what he thought about AI and Machine learning, his response being that he's not interested in things that aren't sound. That really stuck with me, I don't think web scraping is sound, it's guessing retrieval at broken fragments. I admired The Old Reader for including pictures in it's feed, then I saw a CNN post on there about he new M2 Macbook.....with a picture of a kitchen blender as the headline, missing the 5 or so images in the link of M2 Macbooks.
- warrenm 4y agoWhen I've done it in the past, I've just attached the current timestamp of when the item's checked as the pubdate It might be wrong, but it's usually pretty dang close to right (presuming the RSS feed isn't brand-new to the reader)
- tacone 4y agoLooking at it from another point of view: should a RSS reader trust pubDate?
- pipeline_peak 4y agoWhat better source is there?
- tacone 4y agoYou could just display the feed items sorted by the time you fetched them, optionally displaying an informative only date for the entries with pubDate.
- kevincox 4y agoId be surprised if many readers did something besides default to the current time if a date wasn't specified.
- dive 4y agoNetNewsWire uses [0] ‘dateArrived’ as a fallback option. As an example. [0] https://github.com/Ranchero-Software/NetNewsWire/blob/941342f12221880753cd7e426cf6fa59d31029e5/Technotes/ArticlesAndStatuses.markdown#strange-behavior https://github.com/Ranchero-Software/NetNewsWire/blob/941342...
- pipeline_peak 4y agoThanks, there doesn’t seem to be a description of dataArrived
- toomuchtodo 4y agoQuery the Internet Archive’s CDX server for this info. Consider caching the data for performance and good netizen reasons. https://github.com/internetarchive/wayback/blob/master/wayback-cdx-server/README.md https://github.com/internetarchive/wayback/blob/master/wayba...
- pipeline_peak 4y agoThis is really nice, thank you very much!