6 ms·

This is a weird article because it points out that queues don’t solve overload but neither do load shedding or back pressure. All 3 techniques are just differe

This is a weird article because it points out that queues don’t solve overload but neither do load shedding or back pressure. All 3 techniques are just different trade offs on what to do in the face of overload. All 3 have negative ramifications for the users of the system. Load shedding reduces availability, back pressure increases complexity and queues increase latency. In “critical” systems you need all 3. And all 3 can be overloaded. Frankly, your load shedding or back pressure system is probably implemented on a queue one layer down the abstraction.

robertlagrant 3y ago
> Frankly, your load shedding or back pressure system is probably implemented on a queue one layer down the abstraction. If you're building an API, you don't care how your clients cope with your backpressure. You just want to avoid overloading your services, and queues will indeed not help you with that past a certain point, whereas backpressure probably will. Or at least you can scale backpressure much more cheaply than you can scale queues.
- lanstin 3y ago
  Teach your clients to handle 429 with gradual back off/retries and it works well. Even teach your web UI to degrade under partial failure of widget loads.
dijit 3y ago
I think it's the threshold of all people to truly understand that there's no solution to certain problems, only tradeoffs. Queues are great, but can lead to catastrophic failure if you don't have a good way of handling the queue, so making an active choice about how you handle overload is part of designing a robust and resilient system. Trading off new requests for current requests is, in my experience, a valid strategy for eCommerce for example. We called it "quenching".
- kqr 3y ago
  > Trading off new requests for current requests is, in my experience, a valid strategy for eCommerce for example. We called it "quenching". I'm not sure in which direction the trade happens but it sounds like you're dropping older requests in favour of newer. I agree, this has worked well for me also. Surprisingly often the oldest item in the queue is the one for which service will be least valuable.
  - hinkley 3y ago
    I think the theory is that if half of your excess requests come from two actors, then half of the consequences of random dropping are felt by those two people. That’s not fair, but it’s better than dropping all requests after you get into trouble. Fairness is very difficult to achieve in a distributed system, due to coordination delays.
- smaudet 3y ago
  I think part of the article is just the author venting that people make design decisions without understanding the ramifications. Operating a queue as a buffer would be absolutely fine if there were also service level agreements implicit in every queue in operation - i.e. one queue reaches half capacity or is experiencing explosive growth, leading to reactive techniques such as spinning up larger queues/servers, alerting stakeholders to the performance issue(s), and possibly even dynamically rate-limiting end-users. But this is a whole-system centric view of the design. What they're specifically bemoaning is the asinine focus on component-centric design - your widget/service performs slowly? Throw a queue at it (and wonder why you broke everything 100 features and 10x customer base later)! For legitimately small problems, this is OK. But then, you accept that you aren't scaling, and you bake it into your design assumptions. And you sell that, explicitly, to the customer "hey this will only work for 1 thousand/1 million customers, if you want more we can talk".
0xbadcafebee 3y ago
The only real solution to overload (that is, the eventuality of the system not having enough capacity), in modern systems, is autoscaling. Nobody seems to talk about this, I guess because it's taken for granted? But you can literally just keep adding capacity now. We didn't really have that before the cloud; you had the servers you bought, and maybe you'd rush to repurpose some servers to add capacity. Now an algorithm just adds servers magically when your load gets high. Of course the system still has limits. But if you go from zonal to regional to global autoscaling, and you architect specifically to scale up each point in the system using autoscaling, it kind of doesn't have a limit? (that any one product would hit, at a global level) In the past I have spun up a duplicate distributed system in another region and just split the traffic between the two. IaC is crazy.
- secondcoming 3y ago
  That's fine until the issue lies with something that your auto-scaled instances talk to, e.g. Redis, Scylla, SQL DB. There are situations where auto-scaling to infinity makes things far worse.
  - tonyarkles 3y ago
    100% that was my first thought when reading the GP comment. Autoscaling isn't a magic fix. Just like the article says, you need to find the red arrow first to figure out where the bottleneck is and whether or not that bottleneck is actually something within the auto-scaling context or not. You've pointed out a couple of good examples of potential bottlenecks. Another possibility is a downstream 3rd-party service. If you start auto-scaling because a downstream service is struggling, you're just going to end up hitting it even harder and essentially contributing to a DoS attack against them. No silver bullets and all that. Still need to do engineering analysis to figure out what's actually happening before you start shooting.
    - lanstin 3y ago
      Priority queues for your downstream calls to rate limit them. A large enough cluster can take down most any data store, except maybe spanner. Even then you could increase latency a lot while it scales up.
marcosdumay 3y ago
No, load shedding and back pressure present you trade-offs to deal with an overloaded system. Queues don't. Queues just present you problems. If you take an overloaded system and add a queue, every single feature either gets worse or doesn't get any better. And people like to deny this, and pretend that queues will help. They absolutely help with a lot of things but they do nothing but harm in front of an overloaded system.
- naasking 3y ago
  Your mistake is characterising all overloads as the same. Adding a queue to a system that is consistently overloaded won't solve the overload, but many overloads are temporary/bursty, like the Slashdot/HN effect. Queues absolutely do solve these kinds of overloads simply by increasing latency, assuming increased latency is an acceptable choice in your context, of course.
  - 10000truths 3y ago
    Maximum acceptable response times for websites is a few seconds. The HN/Slashdot effect lasts for minutes to hours. That time scale is way too long for a queue to be effective.
    - naasking 3y ago
      You're missing the point. We're talking about general systems here, not websites specifically, and the Slashdot effect is a perfect example that everyone is familiar with where queues do solve maintain availability if longer latency is acceptable.
      - emn13 3y ago
        Furthermore, even as a website aiming to survive some momentary spike in performance - slowing everything down for everyone _can_ be part of a solution to serve more people but fewer things each. People might not normally be willing to wait for more than a second or two for a load; but when they expect or have a sign that things are slower than normal, they might have a little more patience (or just come back to that tab later). I think people are a little too negative on queues. Sure, really naively implemented with entirely unbounded capacity they're an accident waiting to happen... but you don't have to do that.
epr 3y ago
Pretty ironic, right? The entire article is the process of the author making the same mistake he's complaining about. For anyone reading the comments looking for more helpful guidance, the most generally helpful advice you will get is to start by measuring. Your measurements should help you identify bottlenecks, but the answer of how to fix the issue is dependent on your problem's constraints. In fact, there's such a massive number of dimensions to the solution space of the general problem of overloaded systems/services that it is actually ridiculous for the author to try and write a single ~2k words article explaining how to solve them all. If you're hosting some non-critical service (which is clearly the case for the author), then maybe load-shedding is appropriate. If you're making a critical control system on the other hand, load-shedding (allowing system failure by design) is absurd. If you are already set up well for scaling (can be a big up-front investment) and have more money than time, maybe horizontal scaling is a no-brainer. All those examples ignore the elephant in the room, which is the single root cause performance problem you're likely to discover after measuring your system and analyzing the results (good analysis is equally important to measurement, especially in large and/or distributed systems). This can range from something that is very expensive and possibly impractical to fix (ie: entire backend written in python), to a simple tweak (ie: fixing a bad database query) that drastically improves system performance with no downsides. The degree to which a system and it's components were designed by engineers not carelessly throwing away performance at the altar of "premature optimization is the root of all evil" can also have a profound impact.
- lanstin 3y ago
  Your system has a capacity and limits. They can either be explicitly configured in with deliberate load shedding behavior or implicit with a devil may care behavior. Some where between not getting dial tone when you pick up the phone to weird vocal distortions when the limits are hit, choose your poison. Or let randomness choose.

robertlagrant 3y ago

> Frankly, your load shedding or back pressure system is probably implemented on a queue one layer down the abstraction. If you're building an API, you don't care how your clients cope with your backpressure. You just want to avoid overloading your services, and queues will indeed not help you with that past a certain point, whereas backpressure probably will. Or at least you can scale backpressure much more cheaply than you can scale queues.

lanstin 3y ago
Teach your clients to handle 429 with gradual back off/retries and it works well. Even teach your web UI to degrade under partial failure of widget loads.

dijit 3y ago

I think it's the threshold of all people to truly understand that there's no solution to certain problems, only tradeoffs. Queues are great, but can lead to catastrophic failure if you don't have a good way of handling the queue, so making an active choice about how you handle overload is part of designing a robust and resilient system. Trading off new requests for current requests is, in my experience, a valid strategy for eCommerce for example. We called it "quenching".

kqr 3y ago
> Trading off new requests for current requests is, in my experience, a valid strategy for eCommerce for example. We called it "quenching". I'm not sure in which direction the trade happens but it sounds like you're dropping older requests in favour of newer. I agree, this has worked well for me also. Surprisingly often the oldest item in the queue is the one for which service will be least valuable.
- hinkley 3y ago
  I think the theory is that if half of your excess requests come from two actors, then half of the consequences of random dropping are felt by those two people. That’s not fair, but it’s better than dropping all requests after you get into trouble. Fairness is very difficult to achieve in a distributed system, due to coordination delays.
smaudet 3y ago
I think part of the article is just the author venting that people make design decisions without understanding the ramifications. Operating a queue as a buffer would be absolutely fine if there were also service level agreements implicit in every queue in operation - i.e. one queue reaches half capacity or is experiencing explosive growth, leading to reactive techniques such as spinning up larger queues/servers, alerting stakeholders to the performance issue(s), and possibly even dynamically rate-limiting end-users. But this is a whole-system centric view of the design. What they're specifically bemoaning is the asinine focus on component-centric design - your widget/service performs slowly? Throw a queue at it (and wonder why you broke everything 100 features and 10x customer base later)! For legitimately small problems, this is OK. But then, you accept that you aren't scaling, and you bake it into your design assumptions. And you sell that, explicitly, to the customer "hey this will only work for 1 thousand/1 million customers, if you want more we can talk".

0xbadcafebee 3y ago

The only real solution to overload (that is, the eventuality of the system not having enough capacity), in modern systems, is autoscaling. Nobody seems to talk about this, I guess because it's taken for granted? But you can literally just keep adding capacity now. We didn't really have that before the cloud; you had the servers you bought, and maybe you'd rush to repurpose some servers to add capacity. Now an algorithm just adds servers magically when your load gets high. Of course the system still has limits. But if you go from zonal to regional to global autoscaling, and you architect specifically to scale up each point in the system using autoscaling, it kind of doesn't have a limit? (that any one product would hit, at a global level) In the past I have spun up a duplicate distributed system in another region and just split the traffic between the two. IaC is crazy.

secondcoming 3y ago
That's fine until the issue lies with something that your auto-scaled instances talk to, e.g. Redis, Scylla, SQL DB. There are situations where auto-scaling to infinity makes things far worse.
- tonyarkles 3y ago
  100% that was my first thought when reading the GP comment. Autoscaling isn't a magic fix. Just like the article says, you need to find the red arrow first to figure out where the bottleneck is and whether or not that bottleneck is actually something within the auto-scaling context or not. You've pointed out a couple of good examples of potential bottlenecks. Another possibility is a downstream 3rd-party service. If you start auto-scaling because a downstream service is struggling, you're just going to end up hitting it even harder and essentially contributing to a DoS attack against them. No silver bullets and all that. Still need to do engineering analysis to figure out what's actually happening before you start shooting.
  - lanstin 3y ago
    Priority queues for your downstream calls to rate limit them. A large enough cluster can take down most any data store, except maybe spanner. Even then you could increase latency a lot while it scales up.

marcosdumay 3y ago

No, load shedding and back pressure present you trade-offs to deal with an overloaded system. Queues don't. Queues just present you problems. If you take an overloaded system and add a queue, every single feature either gets worse or doesn't get any better. And people like to deny this, and pretend that queues will help. They absolutely help with a lot of things but they do nothing but harm in front of an overloaded system.

naasking 3y ago
Your mistake is characterising all overloads as the same. Adding a queue to a system that is consistently overloaded won't solve the overload, but many overloads are temporary/bursty, like the Slashdot/HN effect. Queues absolutely do solve these kinds of overloads simply by increasing latency, assuming increased latency is an acceptable choice in your context, of course.
- 10000truths 3y ago
  Maximum acceptable response times for websites is a few seconds. The HN/Slashdot effect lasts for minutes to hours. That time scale is way too long for a queue to be effective.
  - naasking 3y ago
    You're missing the point. We're talking about general systems here, not websites specifically, and the Slashdot effect is a perfect example that everyone is familiar with where queues do solve maintain availability if longer latency is acceptable.
    - emn13 3y ago
      Furthermore, even as a website aiming to survive some momentary spike in performance - slowing everything down for everyone _can_ be part of a solution to serve more people but fewer things each. People might not normally be willing to wait for more than a second or two for a load; but when they expect or have a sign that things are slower than normal, they might have a little more patience (or just come back to that tab later). I think people are a little too negative on queues. Sure, really naively implemented with entirely unbounded capacity they're an accident waiting to happen... but you don't have to do that.

epr 3y ago

Pretty ironic, right? The entire article is the process of the author making the same mistake he's complaining about. For anyone reading the comments looking for more helpful guidance, the most generally helpful advice you will get is to start by measuring. Your measurements should help you identify bottlenecks, but the answer of how to fix the issue is dependent on your problem's constraints. In fact, there's such a massive number of dimensions to the solution space of the general problem of overloaded systems/services that it is actually ridiculous for the author to try and write a single ~2k words article explaining how to solve them all. If you're hosting some non-critical service (which is clearly the case for the author), then maybe load-shedding is appropriate. If you're making a critical control system on the other hand, load-shedding (allowing system failure by design) is absurd. If you are already set up well for scaling (can be a big up-front investment) and have more money than time, maybe horizontal scaling is a no-brainer. All those examples ignore the elephant in the room, which is the single root cause performance problem you're likely to discover after measuring your system and analyzing the results (good analysis is equally important to measurement, especially in large and/or distributed systems). This can range from something that is very expensive and possibly impractical to fix (ie: entire backend written in python), to a simple tweak (ie: fixing a bad database query) that drastically improves system performance with no downsides. The degree to which a system and it's components were designed by engineers not carelessly throwing away performance at the altar of "premature optimization is the root of all evil" can also have a profound impact.

lanstin 3y ago
Your system has a capacity and limits. They can either be explicitly configured in with deliberate load shedding behavior or implicit with a devil may care behavior. Some where between not getting dial tone when you pick up the phone to weird vocal distortions when the limits are hit, choose your poison. Or let randomness choose.