3 ms·

> A fair number of people are sending conditional requests, but are doing it every 5 or 10 minutes. This is ridiculous. I don't write that often, and never have

> A fair number of people are sending conditional requests, but are doing it every 5 or 10 minutes. This is ridiculous. I don't write that often, and never have. This objection might make more sense if the purpose of a feed reader was to do nothing except check rachelbythebay.com for updates.

komadori 2y ago
At least at the point you receive a 429, the Retry-After header provides some guidance as to how frequently the resource should be polled.
supriyo-biswas 2y ago
There’s websub/PSHB[1], though I don’t know how widely it’s implemented. [1] https://en.m.wikipedia.org/wiki/WebSub https://en.m.wikipedia.org/wiki/WebSub
- onli 2y ago
  I think that's the right answer. At least for server based readers it seemed to be supported last I checked, and it's the most efficient way to solve the feed deliver with minimal resources on the server.
veeti 2y ago
I requested the feed in my browser _twice_ and already received a 429 with a Retry-After of 24 hours! Not only is this easy to trip by accident, the problem for a feed reader is that it can't know when the author publishes new posts. If a reader happens to scrape the blog 5 minutes before a new entry is posted, it would then take over 23 hours for it to update. There are surely a lot of badly behaved clients out there and I sympathize with that, but the author's policies seem rather heavy handed and counterproductive to providing an open feed. God forbid their most dedicated readers receive the most timely updates.
- masklinn 2y ago
  > If a reader happens to scrape the blog 5 minutes before a new entry is posted, it would then take over 23 hours for it to update. Given the update frequency and kind of content it does not seem much of an issue does it? > the author's policies seem rather heavy handed and counterproductive to providing an open feed It’s not though? “Open feed” is just that, it’s not a license to be a resource hog because you refuse to be a good citizen. > God forbid their most dedicated readers receive the most timely updates. Rachel does not post day to day life updates. “Most timely updates” is not a thing, it’s not a storm reporting feed. Latency is not a concern.
  - veeti 2y ago
    Whether latency is a concern or not depends entirely on the end user controlling the feed reader and their expectations. For all you know, the developer of the reader may be facing complaints from a disgruntled user upset that the feed isn't updating, even though they can see a new blog post on the website. A lecture on HTTP 429 status codes, Retry-After headers and "being a good citizen" is unlikely to make them any happier. You are free to not serve such requests, but as a blogger you might want to consider that such bot traffic serves your legitimate readers. It appears to me that the author is also rate limiting well behaved clients that are already trying to be respectful of resources using conditional requests.
    - simoncion 2y ago
      > A lecture on HTTP 429 status codes, Retry-After headers and "being a good citizen" is unlikely to make them any happier. Why install a lecture in the feed reader? Just have a banner/toast/status-bar-message/whatever that says "This feed will update in X minutes/hours/days.".
      - thaumasiotes 2y ago
        Nothing makes the user happier than telling them they can't use the only functionality you provide. If you're going to show that message, why are you a feed reader at all? Open the webpage and show the content.
noncoml 2y ago
In general it’s not a good argument, as the 10 min polling interval technically controls the latency and has very little to do with the posting frequency or interval between posts.
masklinn 2y ago
As far as I’m concerned it makes more sense if it’s checking even more stuff. If I have 500 feeds in my reader, most of whom update weekly at most, I would assume the reader does something smarter that mindlessly launch 500 requests every 5 minutes.
- vsnf 2y ago
  > I would assume the reader does something smarter that mindlessly launch 500 requests every 5 minutes. Aside from respecting any 429 headers, what is a reader supposed to do without custom update heuristics for each feed that would be considered more than mindless?
  - spc476 2y ago
    Every feed I'm aware of contain multiple entries, so perhaps average the time between posts as a baseline to check? Perhaps add an upper limit to one/hour and a lower limit to one/week? That way, a person who starts posting more frequently will be checked more frequently, and one who slacks off is checked less frequently?
  - mariusor 2y ago
    The least a client can do is respect the HTTP caching headers that Rachel mentions multiple times in her post. RSS has an additional ttl property that can be used.
boricj 2y ago
On my blog, my feed.xml is currently 62654 bytes compressed, 307726 bytes long uncompressed. The compressed version is served with gzip_static, so if the client sets the HTTP header "Accept-Encoding: gzip" then it's sent at no additional CPU cost to me. Looking at my access logs, some RSS readers are well-behaved and get served 304s. Some of them however not only request feed.xml without caching, but they don't even set the HTTP header "Accept-Encoding: gzip". They download feed.xml every time and they download it uncompressed. I'm hosting my blog on a feeble 5-year old Synology NAS located in my apartment. It's not seeing a lot of traffic so it's currently not a problem, but these RSS readers are wasting several orders of magnitude more of network bandwidth and CPU resources (gotta encrypt it for HTTPS) on my end than well-behaved RSS readers. It's just plain embarrassing.
kevincox 2y ago
Exactly, unless she expects every reader to configure the polling interval for her blog this isn't a good solution. I would highly recommend she sets `Cache-Control: max-age=3600` or something similar. That is at least a method that can automatically be applied. Sure, right now most readers just use fixed intervals, but further support for this is a standard that can be pushed slowly. Plus some readers probably use HTTP libraries that would automatically cache the response so some clients will accidentally have support.

komadori 2y ago

At least at the point you receive a 429, the Retry-After header provides some guidance as to how frequently the resource should be polled.

supriyo-biswas 2y ago

There’s websub/PSHB[1], though I don’t know how widely it’s implemented. [1] https://en.m.wikipedia.org/wiki/WebSub https://en.m.wikipedia.org/wiki/WebSub

onli 2y ago
I think that's the right answer. At least for server based readers it seemed to be supported last I checked, and it's the most efficient way to solve the feed deliver with minimal resources on the server.

veeti 2y ago

I requested the feed in my browser _twice_ and already received a 429 with a Retry-After of 24 hours! Not only is this easy to trip by accident, the problem for a feed reader is that it can't know when the author publishes new posts. If a reader happens to scrape the blog 5 minutes before a new entry is posted, it would then take over 23 hours for it to update. There are surely a lot of badly behaved clients out there and I sympathize with that, but the author's policies seem rather heavy handed and counterproductive to providing an open feed. God forbid their most dedicated readers receive the most timely updates.

masklinn 2y ago
> If a reader happens to scrape the blog 5 minutes before a new entry is posted, it would then take over 23 hours for it to update. Given the update frequency and kind of content it does not seem much of an issue does it? > the author's policies seem rather heavy handed and counterproductive to providing an open feed It’s not though? “Open feed” is just that, it’s not a license to be a resource hog because you refuse to be a good citizen. > God forbid their most dedicated readers receive the most timely updates. Rachel does not post day to day life updates. “Most timely updates” is not a thing, it’s not a storm reporting feed. Latency is not a concern.
- veeti 2y ago
  Whether latency is a concern or not depends entirely on the end user controlling the feed reader and their expectations. For all you know, the developer of the reader may be facing complaints from a disgruntled user upset that the feed isn't updating, even though they can see a new blog post on the website. A lecture on HTTP 429 status codes, Retry-After headers and "being a good citizen" is unlikely to make them any happier. You are free to not serve such requests, but as a blogger you might want to consider that such bot traffic serves your legitimate readers. It appears to me that the author is also rate limiting well behaved clients that are already trying to be respectful of resources using conditional requests.
  - simoncion 2y ago
    > A lecture on HTTP 429 status codes, Retry-After headers and "being a good citizen" is unlikely to make them any happier. Why install a lecture in the feed reader? Just have a banner/toast/status-bar-message/whatever that says "This feed will update in X minutes/hours/days.".
    - thaumasiotes 2y ago
      Nothing makes the user happier than telling them they can't use the only functionality you provide. If you're going to show that message, why are you a feed reader at all? Open the webpage and show the content.

noncoml 2y ago

In general it’s not a good argument, as the 10 min polling interval technically controls the latency and has very little to do with the posting frequency or interval between posts.

masklinn 2y ago

As far as I’m concerned it makes more sense if it’s checking even more stuff. If I have 500 feeds in my reader, most of whom update weekly at most, I would assume the reader does something smarter that mindlessly launch 500 requests every 5 minutes.

vsnf 2y ago
> I would assume the reader does something smarter that mindlessly launch 500 requests every 5 minutes. Aside from respecting any 429 headers, what is a reader supposed to do without custom update heuristics for each feed that would be considered more than mindless?
- spc476 2y ago
  Every feed I'm aware of contain multiple entries, so perhaps average the time between posts as a baseline to check? Perhaps add an upper limit to one/hour and a lower limit to one/week? That way, a person who starts posting more frequently will be checked more frequently, and one who slacks off is checked less frequently?
- mariusor 2y ago
  The least a client can do is respect the HTTP caching headers that Rachel mentions multiple times in her post. RSS has an additional ttl property that can be used.

boricj 2y ago

On my blog, my feed.xml is currently 62654 bytes compressed, 307726 bytes long uncompressed. The compressed version is served with gzip_static, so if the client sets the HTTP header "Accept-Encoding: gzip" then it's sent at no additional CPU cost to me. Looking at my access logs, some RSS readers are well-behaved and get served 304s. Some of them however not only request feed.xml without caching, but they don't even set the HTTP header "Accept-Encoding: gzip". They download feed.xml every time and they download it uncompressed. I'm hosting my blog on a feeble 5-year old Synology NAS located in my apartment. It's not seeing a lot of traffic so it's currently not a problem, but these RSS readers are wasting several orders of magnitude more of network bandwidth and CPU resources (gotta encrypt it for HTTPS) on my end than well-behaved RSS readers. It's just plain embarrassing.

kevincox 2y ago

Exactly, unless she expects every reader to configure the polling interval for her blog this isn't a good solution. I would highly recommend she sets `Cache-Control: max-age=3600` or something similar. That is at least a method that can automatically be applied. Sure, right now most readers just use fixed intervals, but further support for this is a standard that can be pushed slowly. Plus some readers probably use HTTP libraries that would automatically cache the response so some clients will accidentally have support.