3 ms·
No, load shedding and back pressure present you trade-offs to deal with an overloaded system. Queues don't. Queues just present you problems. If you take an ov
by marcosdumay 3y ago
No, load shedding and back pressure present you trade-offs to deal with an overloaded system.
Queues don't. Queues just present you problems. If you take an overloaded system and add a queue, every single feature either gets worse or doesn't get any better.
And people like to deny this, and pretend that queues will help. They absolutely help with a lot of things but they do nothing but harm in front of an overloaded system.
- naasking 3y agoYour mistake is characterising all overloads as the same. Adding a queue to a system that is consistently overloaded won't solve the overload, but many overloads are temporary/bursty, like the Slashdot/HN effect. Queues absolutely do solve these kinds of overloads simply by increasing latency, assuming increased latency is an acceptable choice in your context, of course.
- 10000truths 3y agoMaximum acceptable response times for websites is a few seconds. The HN/Slashdot effect lasts for minutes to hours. That time scale is way too long for a queue to be effective.
- naasking 3y agoYou're missing the point. We're talking about general systems here, not websites specifically, and the Slashdot effect is a perfect example that everyone is familiar with where queues do solve maintain availability if longer latency is acceptable.
- emn13 3y agoFurthermore, even as a website aiming to survive some momentary spike in performance - slowing everything down for everyone _can_ be part of a solution to serve more people but fewer things each. People might not normally be willing to wait for more than a second or two for a load; but when they expect or have a sign that things are slower than normal, they might have a little more patience (or just come back to that tab later). I think people are a little too negative on queues. Sure, really naively implemented with entirely unbounded capacity they're an accident waiting to happen... but you don't have to do that.
- lanstin 3y agoThe distinction is really not queues or not but unbounded in theory or not. There are queues everywhere, connection pools, unparsed messages, packets in flight, memory for calls sent out and not yet replied or timed out, etc. but one unbounded (in theory; they are never unbounded in practice) queue can make your system go from robust to fragile.
- marcosdumay 3y agoAfter the overload fixes itself somehow, a queue absolutely solves the availability problem. Yep. If the overload doesn't last for too long. What isn't happening on your example is a queue improving anything on a overloaded system. Queues are immensely helpful for all kinds of problems. Just not for overload.
- naasking 3y agoOverload is only an issue because availability is critical. When it isn't critical, then avoiding overload is obvious and practically trivial.
- lanstin 3y agoAn unbounded queue means you are allocating a lot of stuff during the time you want to cap end all resources handling traffic. It means you slow down when busier. Depending on the framework it often means the system can't recover automatically but will crash if the queue is in memory or need DBAs to flush the queues or whatever.
- multimoon 3y agoAs another user pointed out, this is entirely based on the duration of the overload. If the overload is short, then queues absolutely do help smooth this over for the user. If demand is consistently exceeding capacity, the system will fail regardless which method is used to compensate (queues, shedding, back pressure, etc). Saying that a queues as a whole serve no purpose in an overload is only considering the catastrophic scenario, there is a lot more types of overloads that can happen depending on the system.
- smaudet 3y agoQueues provide you breathing room to increase your throughput. They don't fix the clog, but they keep the kitchen sink from over-flowing while you call the plumber/install a new industrial line.
- lanstin 3y agoA bounded queue that quickly errors back to the caller when full is a fine load shedding system. Unbounded queues are death.
- tanveergill 3y agoI agree that queues can cause problems especially when misconfigured. But some amount of queuing is necessary, to absorb short spikes in demand vs capacity. Also, queues can be helpful to re-order requests based on criticality which won't be possible with zero queue size - in which case we have to immediately drop a request or admit it without considering it's priority. I think it is beneficial to re-think how we tune queues. Instead of setting a queue size, we should be tuning the max permissible latency in the queue which is what a request timeout actually is. That way, you stay within the acceptable response time SLA while keeping only the serve-able requests in the queue. Aperture, an open-source load management platform took this approach. Each request specifies a timeout for which it is willing to stay in the queue. And weighted fair queuing scheduler then allocates the capacity (a request quota or max number of in-flight requests) based on the priority and tokens (request heaviness) of each request. Read more about the WFQ scheduler in Aperture: https://docs.fluxninja.com/concepts/scheduler https://docs.fluxninja.com/concepts/scheduler Link to Aperture's GitHub: https://github.com/fluxninja/aperture https://github.com/fluxninja/aperture Would love to hear your thoughts on our approach!