4 ms·
Suppose I have a feed with a million items already on it which I've already handled. I'm about to poll the feed and there's a small number of new items on the f
by billpg 3y ago
Suppose I have a feed with a million items already on it which I've already handled. I'm about to poll the feed and there's a small number of new items on the feed I don't know about yet.
Am I going to have to download a file with references to all one million items again?
If the answer is to have a separate feed with only the most recent n items, I'm afraid that's not going to work unless there's additional details. There might have been n+1 items turn up since the last time I polled. Also, someone else might want to start handling those one million records from the start and we should be able to co-ordinate by having a common canonical URL.
(I'd also mention pull-vs-push, but that's a thread elsewhere in these comments.)
- deleted 3y ago[deleted]
- bhaak 3y agoEverything will have problems if you start from scratch and have to import millions of items. But of course there's a standard for paging: https://www.rfc-editor.org/rfc/rfc5005#section-3 https://www.rfc-editor.org/rfc/rfc5005#section-3 Then both ATOM and RSS can leverage HTTP headers. If you include If-Modified-Since into your request the server can decide to only returns items that are newer.
- billpg 3y ago<link rel="next" href="http://example.org/index.atom?page=2"/> http://example.org/index.atom?page=2"/> That's fantastic. Anyone developing ATOM/RSS, please use this mechanism. (I'm looking at you, podcast with over 100 episodes.)
- kevincox 3y agoThe problem is that support for this is rare. I know that Podcast Addict supports it but it is rare in podcatchers and even more rare in feed readers. But I definitely agree that it should be a standard feature!
- account42 3y ago> Then both ATOM and RSS can leverage HTTP headers. If you include If-Modified-Since into your request the server can decide to only returns items that are newer. That might work ... most of the time. But its a really ugly layer violation not to mention incompatible with any kind of caching proxy. Don't do this, please.
- deleted 3y ago[deleted]