3 ms·

I requested the feed in my browser _twice_ and already received a 429 with a Retry-After of 24 hours! Not only is this easy to trip by accident, the problem for

by veeti 2y ago

I requested the feed in my browser _twice_ and already received a 429 with a Retry-After of 24 hours! Not only is this easy to trip by accident, the problem for a feed reader is that it can't know when the author publishes new posts. If a reader happens to scrape the blog 5 minutes before a new entry is posted, it would then take over 23 hours for it to update. There are surely a lot of badly behaved clients out there and I sympathize with that, but the author's policies seem rather heavy handed and counterproductive to providing an open feed. God forbid their most dedicated readers receive the most timely updates.

masklinn 2y ago
> If a reader happens to scrape the blog 5 minutes before a new entry is posted, it would then take over 23 hours for it to update. Given the update frequency and kind of content it does not seem much of an issue does it? > the author's policies seem rather heavy handed and counterproductive to providing an open feed It’s not though? “Open feed” is just that, it’s not a license to be a resource hog because you refuse to be a good citizen. > God forbid their most dedicated readers receive the most timely updates. Rachel does not post day to day life updates. “Most timely updates” is not a thing, it’s not a storm reporting feed. Latency is not a concern.
- veeti 2y ago
  Whether latency is a concern or not depends entirely on the end user controlling the feed reader and their expectations. For all you know, the developer of the reader may be facing complaints from a disgruntled user upset that the feed isn't updating, even though they can see a new blog post on the website. A lecture on HTTP 429 status codes, Retry-After headers and "being a good citizen" is unlikely to make them any happier. You are free to not serve such requests, but as a blogger you might want to consider that such bot traffic serves your legitimate readers. It appears to me that the author is also rate limiting well behaved clients that are already trying to be respectful of resources using conditional requests.
  - simoncion 2y ago
    > A lecture on HTTP 429 status codes, Retry-After headers and "being a good citizen" is unlikely to make them any happier. Why install a lecture in the feed reader? Just have a banner/toast/status-bar-message/whatever that says "This feed will update in X minutes/hours/days.".
    - thaumasiotes 2y ago
      Nothing makes the user happier than telling them they can't use the only functionality you provide. If you're going to show that message, why are you a feed reader at all? Open the webpage and show the content.
      - simoncion 2y ago
        I agree that not showing any message is generally preferable. GP seemed fixated on showing a lecture, which I thought was strictly worse than just showing a short message. > If you're going to show that message, why are you a feed reader at all? What? The feed readers I used showed status messages. Status messages are nice, they give the operator a fighting chance to understand what's going on and fix things if they've gone wrong.
- eviks 2y ago
  > not a license to be a resource hog because you refuse to be a good citizen. Which resources are you "hogging" by reloading a webpage twice?
quectophoton 2y ago
> I requested the feed in my browser _twice_ and already received a 429 with a Retry-After of 24 hours! My guess: No `Cache-Control` response header. checks Yup, no `Cache-Control` response header. So she doesn't want more than one request per day, but also doesn't want to add this header to the responses: Cache-Control: public, max-age=86400, stale-if-error=86400
- thaumasiotes 2y ago
  Well, come on. It's your responsibility as a client to check for fresh content in a manner that takes her posting patterns into account. It's not her responsibility as a server to provide content in a manner that takes her posting patterns into account.
  - quectophoton 2y ago
    Serious or not (sounded like sarcasm but one can never be too sure!), I'll elaborate on this point. I think it's both parties' responsibility. Well, "responsibility" may be a strong word, but it depends what you want to achieve. It's okay to return 429 if you receive more than one request per day. But if the goal is to not get those requests in the first place, you can make it way easier with that Cache-Control header. A client might not support the Retry-After header; after all, it's not widely supported by web browsers, so I can see how some libraries might not even bother with adding that maintenance burden. But Cache-Control is more widely supported, so it's more likely to be already implemented in client libraries (or even caching proxies). If someone doesn't support Retry-After, and also ignores Cache-Control, by all means, return an error. It's probably a misconfiguration of their client library at best, or a broken client at worst. I'm not saying one should bend backwards and put unrealistic effort into stuff like this; what I'm saying is, if the HTTP 429 thingy was an acceptable amount of effort, then the Cache-Control thing should be similar (actually way less) effort and time and should slightly reduce the amount of requests received.
pavel_lishin 2y ago
> the problem for a feed reader is that it can't know when the author publishes new posts. It can certainly guess, based on that retry-after header, that she doesn't post more than once a day, and should respect that.

by veeti 2y ago

masklinn 2y ago

> If a reader happens to scrape the blog 5 minutes before a new entry is posted, it would then take over 23 hours for it to update. Given the update frequency and kind of content it does not seem much of an issue does it? > the author's policies seem rather heavy handed and counterproductive to providing an open feed It’s not though? “Open feed” is just that, it’s not a license to be a resource hog because you refuse to be a good citizen. > God forbid their most dedicated readers receive the most timely updates. Rachel does not post day to day life updates. “Most timely updates” is not a thing, it’s not a storm reporting feed. Latency is not a concern.

veeti 2y ago
Whether latency is a concern or not depends entirely on the end user controlling the feed reader and their expectations. For all you know, the developer of the reader may be facing complaints from a disgruntled user upset that the feed isn't updating, even though they can see a new blog post on the website. A lecture on HTTP 429 status codes, Retry-After headers and "being a good citizen" is unlikely to make them any happier. You are free to not serve such requests, but as a blogger you might want to consider that such bot traffic serves your legitimate readers. It appears to me that the author is also rate limiting well behaved clients that are already trying to be respectful of resources using conditional requests.
- simoncion 2y ago
  > A lecture on HTTP 429 status codes, Retry-After headers and "being a good citizen" is unlikely to make them any happier. Why install a lecture in the feed reader? Just have a banner/toast/status-bar-message/whatever that says "This feed will update in X minutes/hours/days.".
  - thaumasiotes 2y ago
    Nothing makes the user happier than telling them they can't use the only functionality you provide. If you're going to show that message, why are you a feed reader at all? Open the webpage and show the content.
    - simoncion 2y ago
      I agree that not showing any message is generally preferable. GP seemed fixated on showing a lecture, which I thought was strictly worse than just showing a short message. > If you're going to show that message, why are you a feed reader at all? What? The feed readers I used showed status messages. Status messages are nice, they give the operator a fighting chance to understand what's going on and fix things if they've gone wrong.
eviks 2y ago
> not a license to be a resource hog because you refuse to be a good citizen. Which resources are you "hogging" by reloading a webpage twice?

quectophoton 2y ago

> I requested the feed in my browser _twice_ and already received a 429 with a Retry-After of 24 hours! My guess: No `Cache-Control` response header. checks Yup, no `Cache-Control` response header. So she doesn't want more than one request per day, but also doesn't want to add this header to the responses: Cache-Control: public, max-age=86400, stale-if-error=86400

thaumasiotes 2y ago
Well, come on. It's your responsibility as a client to check for fresh content in a manner that takes her posting patterns into account. It's not her responsibility as a server to provide content in a manner that takes her posting patterns into account.
- quectophoton 2y ago
  Serious or not (sounded like sarcasm but one can never be too sure!), I'll elaborate on this point. I think it's both parties' responsibility. Well, "responsibility" may be a strong word, but it depends what you want to achieve. It's okay to return 429 if you receive more than one request per day. But if the goal is to not get those requests in the first place, you can make it way easier with that Cache-Control header. A client might not support the Retry-After header; after all, it's not widely supported by web browsers, so I can see how some libraries might not even bother with adding that maintenance burden. But Cache-Control is more widely supported, so it's more likely to be already implemented in client libraries (or even caching proxies). If someone doesn't support Retry-After, and also ignores Cache-Control, by all means, return an error. It's probably a misconfiguration of their client library at best, or a broken client at worst. I'm not saying one should bend backwards and put unrealistic effort into stuff like this; what I'm saying is, if the HTTP 429 thingy was an acceptable amount of effort, then the Cache-Control thing should be similar (actually way less) effort and time and should slightly reduce the amount of requests received.

pavel_lishin 2y ago

> the problem for a feed reader is that it can't know when the author publishes new posts. It can certainly guess, based on that retry-after header, that she doesn't post more than once a day, and should respect that.