4 ms·
> Including an ETag or Last-Modified header in the body of a request when fetching a feed is a mechanism to tell the server to only return new/modified entries/
by viraptor 4y ago
> Including an ETag or Last-Modified header in the body of a request when fetching a feed is a mechanism to tell the server to only return new/modified entries/items (aka: a changeset) since a specific date.
That's not right, is it? The headers are defined at the http level and with caching layers in between, the endpoint is free to return the current feed - with both new and seen entries. Are there many servers optimising that to a shorter feed?
At the very least, static blogs will not filter the entries - they're serving / not serving the same file, regardless of etag.
- Jackevansevo 4y agoIt's a tad confusing. I believe you're totally correct, this is how those headers should behave. > The headers are defined at the http level and with caching layers in between, the endpoint is free to return the current feed - with both new and seen entries. A lot of feeds behave this way. Especially if it's just a static blog > Are there many servers optimising that to a shorter feed? I've come across a fair few that behave with the semantics I described
- aendruk 4y agoYeah, the whole post is riddled with little technical misunderstandings. It’s nice to see someone working things out in the open though.
- Jackevansevo 4y agoAuthor of the post, could you clarify?
- denton-scratch 4y agoI gave up around paragraph four; the article isn't a technical article, it's a touchy-feely people story about an old, blind man with a very white beard.
- Jackevansevo 4y agoI'm not sure I quite agree with this assessment, but thanks.
- hexane360 4y agoAll three of your last comments have been about the wrong article.
- masklinn 4y ago> Are there many servers optimising that to a shorter feed? That's certainly not the official semantics for these headers, they're validators, so that the server can tell the client nothing changed (304). I would assume overriding these for some sort of pagination would also hinder intermediate caches, though I guess that has become less of an issue now that HTTPS is everywhere, but edge caches performing HTTPS termination might still take this information in account? As far as HTTP caching semantics are concerned, the new version of the resource would replace the old one, and new clients would be served the latest cached version, truncated. In fact the requests headers make that very clear, as they're called respectively If-None-Match and If-Modified-Since. Incidentally there are also If-Match and If-Unmodified-Since headers (POST, PUT, DELETE), but I don't know if anyone actually uses them in the wild. IIRC they were intended for "transactional" update guarantees: you'd fetch a resource, then PUT to it with If-Match and / or If-Unmodified-Since, and you'd a 412 (Precondition Failed) if the resource had been modified in the meantime.
- Animats 4y agoUn-duplicating RSS items is hard. Timestamps can't be trusted. IDs can't be trusted. Some sources will resend the same item with a different timestamp. RSS servers behind a load balancer may return different IDs and timestamps for the same items. I had to compute a hash of each item to reliably remove duplicates. Here's a un-duplicator in a feed reader I wrote back in 2009.[1] This is used for printing RSS feeds on antique teletype machines. Reliable duplicate removal is essential when printing at 5 characters per second. [1] https://github.com/John-Nagle/baudotrss/blob/master/messager/newsfeed.py https://github.com/John-Nagle/baudotrss/blob/master/messager...
- thorio 4y agoTrue, I realized this when developing the initial version of zebra I posted a few days back. However relying on a SQL server that requires a unique URL turned out to be the easiest and most effective solution. https://play.google.com/store/apps/details?id=thorio.solutions.zebra&hl=en_US&gl=US https://play.google.com/store/apps/details?id=thorio.solutio... Beyond that you probably could cover the last 2 or so percent using string comparison against title and description or peppeteering the website. Zebra as a social feed reader was a great learning: for example that a lot of sites circulate their content multiple times in different packages (/tiles) and very few flag paywalled content - still working on recognizing that. Any hints for a good way to distinguish that, when investigating the urls?
- Animats 4y agoI was retrieving text from news sites, so URLS were not that relevant. Some news services will re-issue a story with more information, keeping the same title and description. A full text check is necessary. I computed a secure hash of the text and compared that.
- viraptor 4y agoThere are two extremes i know here: same/similar title changing content (we hit gold in seo, let's keep updating this "10 best foos for baring" page), changing title same content (anyone doing serious A/B testing).