4 ms·

Blocked for 2 hits in 20 minutes on a light protocol like rss? That seems hilariously aggressive to me, but her server her rules I guess.

by Havoc 2y ago

garfij 2y ago
I believe if you read carefully, it's not blocked, it's rate limited to once daily, with very clear remediation steps included in the response.
- that_guy_iain 2y ago
  If you understand what rate limiting is, you block them for a period of time. Let's stop being pedantic here. 72 requests per day is nothing and acting like it's mayhem is a bit silly. And for a lot of people would result in them getting possible news slower. Sure OP won't publish that often but their rate limiting is an edge case and should be treated as such. If they're blocked until the next day and nothing gets updated then the only person harmed is OP for being overly bothered by their HTTP logs. Sure it's their server and they can do whatever they want. But all this does is hurts the people trying to reach their blog.
  - HomeDeLaPot 2y ago
    72 requests per day _per user with a naive feed reader_. This is a small personal blog with no ads that OP is self-hosting on her own hardware, so blocking all this junk traffic is probably saving her money. Plus she's calling attention to how feed readers can be improved!
    - that_guy_iain 2y ago
      Even if they had 1000 feed readers which would be a massive amount for a blog, if you can't scale that cheaply, that's on you. As I pointed out, her blog and rate limiting are an extreme edge case, it would be silly for anyone to put effort into changing their feed reader for a single small blog. It's bad product management.
      - tecleandor 2y ago
        Of course she can. It's static. She doesn't want and I understand. She's signaling their clients an standard call to say "I think you already have read this, at lest ask me first when this changed the last time".
        that_guy_iain 2y ago
        If you choose to run a poorly implemented rss feed and not scale it cheaply you lose any sympathy from me.
        EdwardDiego 2y ago
        So long as you know what If-Modified-Since is, and use it, you can have all or none of the sympathy you want.
    - m3047 2y ago
      My reason for smacking stuff down is that I don't want to see it in my logs. That simple.
  - throw0101b 2y ago
    > 72 requests per day is nothing and acting like it's mayhem is a bit silly. 72 requests per day per IP over how many IPs? When you start multiplying numbers together they can get big.
  - quest88 2y ago
    I invite you to run your own popular blog on your own hardware and pay for the costs. It sounds like you don't know what the true costs are.
    - Twirrim 2y ago
      OP has never said that this is about financial aspects of things.
      - Joker_vD 2y ago
        Yews, it's about enforcing their preference on how others should interact with OP's published site feed, on principle. Which is always an uphill battle.
        Twirrim 2y ago
        It's about enforcing that people follow standards. Which is still an uphill battle, but at least it's based in something sane. Their work on this has resulted in improvements to a whole slew of popular feed readers that should make life easier for a chunk of the internet, not just OP's own site.
    - that_guy_iain 2y ago
      Sounds like you don't know how to scale for cheap. And since I've ran integrations that connected over 500 companies. I know what a rouge client actually looks like and 72 requests per day and I wouldn't even notice.
      - EdwardDiego 2y ago
        Good on you champ.
    - donatj 2y ago
      I do run a popular blog, and a $5 a month Digital Ocean droplet handles millions of requests per month without breaking a sweat.
sccxy 2y ago
Her rss feed is last 100 posts with full content. So it means 30 months of blog posts content in single request. Sending 0.5MB in single rss request is more crime than those 2 hits in 20 minutes.
- horsawlarway 2y ago
  I generally agree here. There are a lot of very valid use cases where defaulting to deny for an entire 24 hour cycle after a single request is incredible frustrating for your downstream users (shared IP at my university means I will never get a non-429 response... And God help me if I'm testing new RSS readers...) It's her server, so do as you please, I guess. But it's a hilariously hostile response compared to just returning less data.
  - mrweasel 2y ago
    > But it's a hilariously hostile response compared to just returning less data. So provide a poor service to everyone, because some people doesn't know how to behave. That sees like an even worse response.
    - sccxy 2y ago
      Send only one year's recent posts and you've reduced bandwidth by 50%.
    - wakawaka28 2y ago
      People don't want to have to customize refresh rates on a per-feed basis. Perhaps the RSS or Atom standards need to support importing the recommended refresh rate automatically.
      - ncallaway 2y ago
        They don't need to change the refresh rate, though. They need to make conditional requests with an etag or a last-modified date, so the server can respond with a 304 not modified if no changes have been made. No standards need to be updated. The client software needs to be a better HTTP citizen.
yladiz 2y ago
That’s a bit disingenuous. 429s aren’t “blocking”, they’re telling the requester that they’re done too many requests and to try again later (with a value in the header). I assume the author configured this because they know how often the site is going to change typically. That the web server eventually stops responding if the client ignores requests isn’t that surprising, but I doubt it was configured directly too.
- that_guy_iain 2y ago
  I would say it's disingenuous to claim sending HTTP status and body that is not expected for a period of time is not blocking them for that period of time. You can be pedantic and claim "but they can still access the server" but in reality that client is blocked for a period of time.
  - kstrauser 2y ago
    In that case, I should be irate that the AWS API blocks me many times per day. Run `aws cli service some-paginated-thing` and see how many retries you get during normal, routine operation. But I’m not, because they’re not blocking me. They’re asking my client to slow down. Neither AWS nor Rachel’s blog owes me unlimited requests per unit time, and neither have “blocked” me when I violate they policies.
    - that_guy_iain 2y ago
      They literally do block you for a period of time until you are out of the rate limit. That is how rate limits work. That's why you don't get to access the resource you requested, because their system literally blocked you from doing so. See when you're trying to be pedantic and all about semantics, you should make sure you've crossed your Ts and dotted your Is. > Block – AWS WAF blocks the request and applies any custom blocking behavior that you've defined. from https://docs.aws.amazon.com/waf/latest/developerguide/waf-rule-statement-type-rate-based-request-limiting.html https://docs.aws.amazon.com/waf/latest/developerguide/waf-ru... And my favourite > Rate limiting blocks users, bots, or applications that are over-using or abusing a web property. Rate limiting can stop certain kinds of bot attacks. From CloudFlare's explainer https://www.cloudflare.com/learning/bots/what-is-rate-limiting/ https://www.cloudflare.com/learning/bots/what-is-rate-limiti... Every documentation on rate limit will include the word block. Because that's what you do, you allow access for a specific amount of requests and then block those that go over.
cesarb 2y ago
> Blocked for 2 hits in 20 minutes on a light protocol like rss? I might be getting old, but 500KB in a single response doesn't feel "light" to me.
- sccxy 2y ago
  Yes, this is a very poorly designed RSS feed. 500KB is horrible for RSS.
  - Symbiote 2y ago
    It's reasonable to have whole articles in RSS, of you aren't trying to show ads or similar.
    - sccxy 2y ago
      Whole articles are reasonable. 100 articles are not reasonable. 100 articles where most of them are 1+ year old is madness. RSS is not an archive of the entire website.
      - sangnoir 2y ago
        > RSS is not an archive of the entire website Whole-article feeds end up become exactly that - a local archive of a blog.
      - Sweepi 2y ago
        Well, its note the entire website, and i find the "last 100 articles" rule way better than "last 3" or "last 90 days" (which some times is 0 or 1). The host is fine with sending 0.5 MiB once (the client should be aswell from both a bandwidth and storage point of view). The host is not fine with sending 0.5 MiB every 20 minutes, which could be easily avoided if the client would use the mentioned "If-Modified-Since header".
        xyzsparetimexyz 2y ago
        Thanks for being someone who actually knows this stuff among all the Dunning Kruger replies
mrweasel 2y ago
But it's not a "light" protocol when you're serving 36MB per day, when 500KB would suffice. RSS/Atom is light weight, if clients play by the rules. This could also have been a news website, imagine how much traffic would be dedicated to pointless transfers of unchanged data. Traffic isn't free. A similar problem arise from the increase in AI scraper activities. Talking to other SREs the problem seems pretty wide spread. AI companies will just hoover up data, but revisit so frequently and aggressively that it's starting to affect the transit feeds for popular websites. Frequently user-agents wouldn't be set to something unique, or deliberately hidden, and traffic originates from AWS, making it hard to target individual bad actors. Fair enough that you're scraping websites, that's part of the game when your online, but when your industry starts to affect transit feeds, then we need to talk compensation.
II2II 2y ago
If your feed reader is refreshing every 20 minutes for a blog that is updated daily, nearly 99% of the data sent is identical. It looks like Rachel's blog is updated (roughly) weekly, so that jumps to 99.8%. It's not the least efficient thing in the world of computers, but it is definitely incurring unnecessary costs.
- elashri 2y ago
  I opened the xml file she provides in the blog and it seems very long but okay. Then I decided it is a good blog to subscribe so I went and tried to add to my freshrss selfhosted instance (same ip obviously) and I couldn't because I got blocked/rate limited. So yes it is aggressive for different reasons.
  - mubou 2y ago
    Yeah, that's insane. Pretty much telling me not to subscribe to your blog at that point. Like sites that have an rss feed yet put Cloudflare protection in front of it... The correct thing to do here is put a caching layer in front so that every feed reader isn't simultaneously hitting the origin for the same content. IP banning is the wrong approach. (Even if it's only a temporary block, that's going to cause my reader to show an error and is entirely unnecessary.)
  - radicality 2y ago
    Weird. Those should have had different user-agents, and I would guess it cannot be purely based on up.
  - KomoD 2y ago
    Same, I made 3 requests in total and got blocked.
  - lilyball 2y ago
    I know she's mentioned this particular problem before on her blog, I don't remember where to find it offhand now but my vague recollection is that because browsers have largely removed the ability to directly view RSS feeds she doesn't consider this a significant issue anymore. Why did you view the XML file directly?

Blocked for 2 hits in 20 minutes on a light protocol like rss? That seems hilariously aggressive to me, but her server her rules I guess.

by Havoc 2y ago

Blocked for 2 hits in 20 minutes on a light protocol like rss? That seems hilariously aggressive to me, but her server her rules I guess.

garfij 2y ago

I believe if you read carefully, it's not blocked, it's rate limited to once daily, with very clear remediation steps included in the response.

that_guy_iain 2y ago
If you understand what rate limiting is, you block them for a period of time. Let's stop being pedantic here. 72 requests per day is nothing and acting like it's mayhem is a bit silly. And for a lot of people would result in them getting possible news slower. Sure OP won't publish that often but their rate limiting is an edge case and should be treated as such. If they're blocked until the next day and nothing gets updated then the only person harmed is OP for being overly bothered by their HTTP logs. Sure it's their server and they can do whatever they want. But all this does is hurts the people trying to reach their blog.
- HomeDeLaPot 2y ago
  72 requests per day _per user with a naive feed reader_. This is a small personal blog with no ads that OP is self-hosting on her own hardware, so blocking all this junk traffic is probably saving her money. Plus she's calling attention to how feed readers can be improved!
  - that_guy_iain 2y ago
    Even if they had 1000 feed readers which would be a massive amount for a blog, if you can't scale that cheaply, that's on you. As I pointed out, her blog and rate limiting are an extreme edge case, it would be silly for anyone to put effort into changing their feed reader for a single small blog. It's bad product management.
    - tecleandor 2y ago
      Of course she can. It's static. She doesn't want and I understand. She's signaling their clients an standard call to say "I think you already have read this, at lest ask me first when this changed the last time".
      - that_guy_iain 2y ago
        If you choose to run a poorly implemented rss feed and not scale it cheaply you lose any sympathy from me.
        EdwardDiego 2y ago
        So long as you know what If-Modified-Since is, and use it, you can have all or none of the sympathy you want.
  - m3047 2y ago
    My reason for smacking stuff down is that I don't want to see it in my logs. That simple.
- throw0101b 2y ago
  > 72 requests per day is nothing and acting like it's mayhem is a bit silly. 72 requests per day per IP over how many IPs? When you start multiplying numbers together they can get big.
- quest88 2y ago
  I invite you to run your own popular blog on your own hardware and pay for the costs. It sounds like you don't know what the true costs are.
  - Twirrim 2y ago
    OP has never said that this is about financial aspects of things.
    - Joker_vD 2y ago
      Yews, it's about enforcing their preference on how others should interact with OP's published site feed, on principle. Which is always an uphill battle.
      - Twirrim 2y ago
        It's about enforcing that people follow standards. Which is still an uphill battle, but at least it's based in something sane. Their work on this has resulted in improvements to a whole slew of popular feed readers that should make life easier for a chunk of the internet, not just OP's own site.
  - that_guy_iain 2y ago
    Sounds like you don't know how to scale for cheap. And since I've ran integrations that connected over 500 companies. I know what a rouge client actually looks like and 72 requests per day and I wouldn't even notice.
    - EdwardDiego 2y ago
      Good on you champ.
  - donatj 2y ago
    I do run a popular blog, and a $5 a month Digital Ocean droplet handles millions of requests per month without breaking a sweat.

sccxy 2y ago

Her rss feed is last 100 posts with full content. So it means 30 months of blog posts content in single request. Sending 0.5MB in single rss request is more crime than those 2 hits in 20 minutes.

horsawlarway 2y ago
I generally agree here. There are a lot of very valid use cases where defaulting to deny for an entire 24 hour cycle after a single request is incredible frustrating for your downstream users (shared IP at my university means I will never get a non-429 response... And God help me if I'm testing new RSS readers...) It's her server, so do as you please, I guess. But it's a hilariously hostile response compared to just returning less data.
- mrweasel 2y ago
  > But it's a hilariously hostile response compared to just returning less data. So provide a poor service to everyone, because some people doesn't know how to behave. That sees like an even worse response.
  - sccxy 2y ago
    Send only one year's recent posts and you've reduced bandwidth by 50%.
  - wakawaka28 2y ago
    People don't want to have to customize refresh rates on a per-feed basis. Perhaps the RSS or Atom standards need to support importing the recommended refresh rate automatically.
    - ncallaway 2y ago
      They don't need to change the refresh rate, though. They need to make conditional requests with an etag or a last-modified date, so the server can respond with a 304 not modified if no changes have been made. No standards need to be updated. The client software needs to be a better HTTP citizen.

yladiz 2y ago

That’s a bit disingenuous. 429s aren’t “blocking”, they’re telling the requester that they’re done too many requests and to try again later (with a value in the header). I assume the author configured this because they know how often the site is going to change typically. That the web server eventually stops responding if the client ignores requests isn’t that surprising, but I doubt it was configured directly too.

that_guy_iain 2y ago
I would say it's disingenuous to claim sending HTTP status and body that is not expected for a period of time is not blocking them for that period of time. You can be pedantic and claim "but they can still access the server" but in reality that client is blocked for a period of time.
- kstrauser 2y ago
  In that case, I should be irate that the AWS API blocks me many times per day. Run `aws cli service some-paginated-thing` and see how many retries you get during normal, routine operation. But I’m not, because they’re not blocking me. They’re asking my client to slow down. Neither AWS nor Rachel’s blog owes me unlimited requests per unit time, and neither have “blocked” me when I violate they policies.
  - that_guy_iain 2y ago
    They literally do block you for a period of time until you are out of the rate limit. That is how rate limits work. That's why you don't get to access the resource you requested, because their system literally blocked you from doing so. See when you're trying to be pedantic and all about semantics, you should make sure you've crossed your Ts and dotted your Is. > Block – AWS WAF blocks the request and applies any custom blocking behavior that you've defined. from https://docs.aws.amazon.com/waf/latest/developerguide/waf-rule-statement-type-rate-based-request-limiting.html https://docs.aws.amazon.com/waf/latest/developerguide/waf-ru... And my favourite > Rate limiting blocks users, bots, or applications that are over-using or abusing a web property. Rate limiting can stop certain kinds of bot attacks. From CloudFlare's explainer https://www.cloudflare.com/learning/bots/what-is-rate-limiting/ https://www.cloudflare.com/learning/bots/what-is-rate-limiti... Every documentation on rate limit will include the word block. Because that's what you do, you allow access for a specific amount of requests and then block those that go over.

cesarb 2y ago

> Blocked for 2 hits in 20 minutes on a light protocol like rss? I might be getting old, but 500KB in a single response doesn't feel "light" to me.

sccxy 2y ago
Yes, this is a very poorly designed RSS feed. 500KB is horrible for RSS.
- Symbiote 2y ago
  It's reasonable to have whole articles in RSS, of you aren't trying to show ads or similar.
  - sccxy 2y ago
    Whole articles are reasonable. 100 articles are not reasonable. 100 articles where most of them are 1+ year old is madness. RSS is not an archive of the entire website.
    - sangnoir 2y ago
      > RSS is not an archive of the entire website Whole-article feeds end up become exactly that - a local archive of a blog.
    - Sweepi 2y ago
      Well, its note the entire website, and i find the "last 100 articles" rule way better than "last 3" or "last 90 days" (which some times is 0 or 1). The host is fine with sending 0.5 MiB once (the client should be aswell from both a bandwidth and storage point of view). The host is not fine with sending 0.5 MiB every 20 minutes, which could be easily avoided if the client would use the mentioned "If-Modified-Since header".
      - xyzsparetimexyz 2y ago
        Thanks for being someone who actually knows this stuff among all the Dunning Kruger replies

mrweasel 2y ago

But it's not a "light" protocol when you're serving 36MB per day, when 500KB would suffice. RSS/Atom is light weight, if clients play by the rules. This could also have been a news website, imagine how much traffic would be dedicated to pointless transfers of unchanged data. Traffic isn't free. A similar problem arise from the increase in AI scraper activities. Talking to other SREs the problem seems pretty wide spread. AI companies will just hoover up data, but revisit so frequently and aggressively that it's starting to affect the transit feeds for popular websites. Frequently user-agents wouldn't be set to something unique, or deliberately hidden, and traffic originates from AWS, making it hard to target individual bad actors. Fair enough that you're scraping websites, that's part of the game when your online, but when your industry starts to affect transit feeds, then we need to talk compensation.

II2II 2y ago

If your feed reader is refreshing every 20 minutes for a blog that is updated daily, nearly 99% of the data sent is identical. It looks like Rachel's blog is updated (roughly) weekly, so that jumps to 99.8%. It's not the least efficient thing in the world of computers, but it is definitely incurring unnecessary costs.

elashri 2y ago
I opened the xml file she provides in the blog and it seems very long but okay. Then I decided it is a good blog to subscribe so I went and tried to add to my freshrss selfhosted instance (same ip obviously) and I couldn't because I got blocked/rate limited. So yes it is aggressive for different reasons.
- mubou 2y ago
  Yeah, that's insane. Pretty much telling me not to subscribe to your blog at that point. Like sites that have an rss feed yet put Cloudflare protection in front of it... The correct thing to do here is put a caching layer in front so that every feed reader isn't simultaneously hitting the origin for the same content. IP banning is the wrong approach. (Even if it's only a temporary block, that's going to cause my reader to show an error and is entirely unnecessary.)
- radicality 2y ago
  Weird. Those should have had different user-agents, and I would guess it cannot be purely based on up.
- KomoD 2y ago
  Same, I made 3 requests in total and got blocked.
- lilyball 2y ago
  I know she's mentioned this particular problem before on her blog, I don't remember where to find it offhand now but my vague recollection is that because browsers have largely removed the ability to directly view RSS feeds she doesn't consider this a significant issue anymore. Why did you view the XML file directly?