19 ms·
No they shouldn’t. The way I see it is that there are two opposing things you can optimise for that depend on queue depth: utilisation or latency. If you care
by orf 4y ago
No they shouldn’t.
The way I see it is that there are two opposing things you can optimise for that depend on queue depth: utilisation or latency.
If you care about processing each message as quickly as possible then queues should be empty. This often requires a higher cost and lower utilisation as you inevitably have idle workers waiting for new messages.
If you care about utilisation, then you never want your queue to be empty while something is polling it. This might be some background task that runs on a GPU - every second that sits idle is wasted cash, and those tasks usually benefit heavily from batching inputs together.
In this case you want it to always have something to read from the queue and shut it down the moment this isn’t the case.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- anonymoushn 4y agoSeems like there's some implicit assumption in this comment that everything is rented by the minute and latency never matters. Renting things by the minute isn't really the cheapest way to buy lots of things though. Also, its sometimes nice to respond to the content of the article instead of its title.
- orf 4y agoI think the implicit assumption in the comment is that it would be possible for readers to understand the underlying idea of the comment rather than only see the specific example. “resource cost per minute” is not the only thing helped by a non-empty queue. Also please don’t accuse someone of not “responding to the content of the article” when the content is a bog standard list. If you want my response to that: “yes, that is indeed a list of 4 things queues help with”. Not sure it adds much to the discussion.
- anonymoushn 4y agoWhen will my users be happier because I took longer to do things on their behalf? I apologize for not understanding the unwritten portion of the comment. I guess I should really ask, why do people care about utilization instead of throughput, latency, and cost? We could increase utilization by rewriting everything to be more wasteful, for example, but it doesn't seem like we should do it.
- zmgsabst 4y agoUtilization is choosing throughout and cost over latency. People care about utilization when, eg, processing petabytes of data — something that takes hours to days, where you’re optimizing batch throughout and cost. By making sure you efficiently utilize your hardware.
- anonymoushn 4y ago> Utilization is choosing throughout and cost over latency. For a given rate of incoming work, increasing throughput decreases utilization. So I guess it's not choosing throughput too much. > People care about utilization when, eg, processing petabytes of data — something that takes hours to days, where you’re optimizing batch throughout and cost. By making sure you efficiently utilize your hardware. If you have fixed resources and lots of work, it would be good to have nonempty queues to prevent them from idling. That is maybe the big unwritten part of the comment upthread. It doesn't seem worth saying "no, the article is wrong in all cases because of this particular case!" though, so I'm not sure.
- remram 4y agoIf you define throughput as "processed items over time per processor", the tradeoff becomes clear. Given the rate of incoming requests, you can get more processors, which will increase cost and reduce latency and throughput-per-processor, or you can get fewer processors, which will decrease cost and increase latency and throughput-per-processor. If you have fixed resources and fixed work, wanting your queues to be empty is a wish, not a strategy.
- Kinrany 4y agoIsn't a full queue basically equivalent to no queue? The same number of requests is being dropped.
- yxhuvud 4y agoNo, there are use cases where new entries are not ingested by requests - sometimes you know all work that needs to be done upfront. And sometimes you have a situation where you have a few requests coming in over time but that you at the same time need to sometimes go through everything you have seen earlier, once more and flush it through the queue.
- WJW 4y agoNot all queues serve (soft) real time requests and not all queues are bounded (1). If the rate of incoming queue items is predictable but not constant (for example, average load in the weekend or at night tends to be lower for many systems) you can often get away with having enough capacity to serve the average load while using the queue as a buffer. This is fine if the work done is not time critical, like sending most types of marketing emails. You might argue that a queue with many items in it which can still accept more items is not really "full", but it is clearly not empty either. (1) Obviously all queues are bounded by the capacity of the hardware, but since most queueing software can persist to disk and/or lives in the cloud with elastic storage this is not usually a problem.
- remram 4y agoThe difference is that the requests that are being served are getting extreme latency. You are serving the same number of requests, but every single one is X days old.
- ghusbands 4y ago> should always be full > always have something to read from the queue In the first case, you've gone a bit far. To maximise utilisation, you want the queue to never be empty when polled.
- orf 4y agoIndeed, and this is what I meant but worded poorly. Obviously you don’t want it to be 100% full, but you do want it to be the kind of full you get after a nice meal but you still have room for desert and a coffee.
- roflyear 4y agoGood reading here: https://blog.danslimmon.com/2016/08/26/the-most-important-thing-to-understand-about-queues/ https://blog.danslimmon.com/2016/08/26/the-most-important-th... You really don't want your queue more than 70-80% utilized. Once it gets to 75% you should consider that "red zone" and you need to fix it somehow. Once the queue is 80% you're really hitting into some issues. Above 80% and you will have significant problems.
- orf 4y agoThanks for this, that has helped me see something I missed. If you’re dealing with something like SQS and are receiving variable sized batches of messages, the consumers can be classed as “100% utilised” (always processing a batch of messages with no empty receives) whilst the queue would not be considered 100% utilised (the average delivered batch size is always less than the max).
- roflyear 4y agoWhat issue are you trying to solve? The expense of polling? Obviously there is a line where you do not want a bunch of expensive consumers hanging around - but this isn't the place IMO to micro-optimize your costs. You shouldn't have like 20x the consumers you need but you shouldn't worry about having EXACTLY the correct amount. In your example, are you implying that if more jobs come in more workers get spawned? Is that specific to the Amazon service?
- kilgnad 4y agoWrong. They should. Any queue that fills up, say halfway, is an indication the system is in a precarious state. Its simple. Queues have limited space if the velocity of data going in is faster then data going out the queue will fill up until it's full. A half full queue indicates the velocity is in such a state and such a state is unsustainable. You need to build your system such that it stays out of unsustainable states as much as possible. If you're finding that your queue fills up half way a lot that means you're building your system in such a way that it operates at the border of sustainability. In other words the system often ingests data at rates that cannot be sustained indefinitely. If a system with a queue half full operates in that same state twice as long, you now have a full queue and you're losing data. In essence your queues should mostly be empty most of the time. At best you can account for occasional spikes as the article stated but a queue even half full should never be the operational norm.
- orf 4y agoThis response lacks nuance and focuses too much on the operational side of queues. Not all queues are “realistically bounded”. Let’s take a hypothetical silly “order queue” where each entry represents a €10 item that’s been purchased. The queue feeds into a boxing machine that puts the item in a box and posts it to the user. The point at which a queue hits an upper bound would represent billions of euros of orders. That’s not really a realistic failure mode to hit without noticing long beforehand. Meanwhile, if you always want your queue empty then your boxing machine is going to be shipping thousands of small boxes because the moment an order arrives, it’s processed into a box. It’s much more efficient to have a small, managed and time-bounded backlog of orders that the boxing machine uses to batch orders together in a smaller number of larger boxes.
- kilgnad 4y agoWrong. Your thinking is where the nuance is lacking. As I said you need to think of the system in terms of it's derivative. The rate of data consumed must on average be faster then the production. The net difference of these rates must equal a negative velocity with the rate of data going in as a positive number and data consumed as negative. There is no system on the face of the earth that can operate at an average positive velocity. That is categorical failure. This is mathematically and logically true no matter how big your queue is. Even for the situation your describing... A really big ass queue, the average velocity of that system must still be negative. Unless your queue is so big that it can withstand an average negative velocity over the lifetime of the system which is kind of absurd. Most of the time your queues should be empty. If your queues are observationally often filled with data but not full it means you're still operating at a negative velocity but you've tuned your system to operate at the border of failure. It means your system hits unsustainable velocities often. You may want this for cost reasons but but such a system is not robust enough for my comfort. The queue should be empty most of the time with occasional spikes at most.
- smetj 4y agoI came here to say the same and saw your comment. Totally agree.
- atdixon 4y agoYou’re assuming the consumer is the one producing batches.
- mabbo 4y agoIf a queue isn't approaching empty, then its input is exceeding its output. Therefore it's size is trending to infinity. Since we don't have infinite space, we can expect eventually to lose some messages in this scenario.
- deleted 4y ago[deleted]
- beardedetim 4y agoI think this is a valid concern for Future Us to worry about. If you have alerts for when this is going to happen, you can punt the issue until it starts to become a concern and who knows if your product will be alive when that issue happens.
- londons_explore 4y agoM/M/1 Queue theory doesn't apply to real queues... Because the work input into the queue usually does depend on the latency of previous work completed by the worker. If there is a 1 hour queue for the checkout at a grocery store, you probably won't join the queue - you'll go to a different store...
- lokar 4y agoIn many systems you get the reverse. A client sends a retry without the ability to cancel the first request, now it has two requests in the queue.
- cryptonector 4y agoBack pressure and circuit breakers matter.
- orf 4y ago> Since we don't have infinite space, we can expect eventually to lose some messages in this scenario. Sure, that's technically correct but applies to basically everything. It is very likely your users/orders/whatever table in your database is also "trending to infinity" over time, except this doesn't mean databases should always be empty or that we should expect to eventually lose some users/orders/whatever. Or, more succinctly, if your queue capacity is 10 million messages and your queue messages represent "houses purchased through our website", then in reality your queue capacity is infinite because nobody will ever purchase 10 million homes through your application per small unit of time.
- roflyear 4y agoIf your queue is always full (100% utilization) your jobs will on average never be processed. Love the downvotes! People should read this: https://blog.danslimmon.com/2016/08/26/the-most-important-thing-to-understand-about-queues/ https://blog.danslimmon.com/2016/08/26/the-most-important-th... I understand this is a little counterintuitive but it is true. You should not have a queue that always has a job ready to be processed.
- adrr 4y agoQueue isn’t full. It isn’t empty and things are being processed not in “real” time. I’ve always run my queues where they are never empty. If queue depth hits certain levels, workers are scaled up. When it drops, they are scaled down. Last thing I want is under utilized instances.
- roflyear 4y agoUnsure what you're saying. If your queue is NEVER empty wait times will grow to infinity. You shouldn't aim for this. You should aim for something reasonable like, "empty 50% of the time" etc..
- adrr 4y agoThey haven’t been empty for two years and my waiting times average 5 minutes and 30 minutes during high load times. I don’t understand how they will grow to infinity.
- roflyear 4y ago> I don’t understand how they will grow to infinity. You either have to be consuming faster than the queue is growing, or you are adding faster than you consume. There is no equilibrium. If you are consuming faster than it is growing, at some point you'll consume all jobs, meaning your queue is sometimes empty. If you are adding faster than you consume, your wait time approaches infinity (tho in reality you'll fill up your queue storage first, of course).
- marcosdumay 4y agoYour comment only shows that if you value resource utilization, you shouldn't use queues. Because, no, nothing you said changes the fact that queues only do anything when kept mostly empty. And the only thing they do is some short term trade of latency for reliability. They actually can't change utilization.
- orf 4y agoI’m currently waiting for a flight. Isn’t airport is a kind of queue that buffers people over time then delivers them to the sky at steady, fixed intervals? This airport seems busy but the planes seem full, how can we improve utilisation?
- Jtsummers 4y agoWhat does it mean to improve utilization if the aircraft are at 100% utilization?
- orf 4y agoBigger aircraft I guess. But the point is that we are at full utilisation despite the “queue” (airport) never being empty, and thousands of people awaiting “delivery”. Which refutes the original point about queues not doing much unless they are empty and being unable to effect utilisation. If you have a somewhat inelastic output that benefits from batching, but has variable inputs, they are absolutely crucial to utilisation.
- Jtsummers 4y agoI'm being nitpicky but my intended prompt was to get you to define what needs to be improved, which you did. You need to improve (or increase) capacity, not utilization. The two are related, but they aren't the same. Utilization being at 100% means it only has two ways to go: Stay at 100% or drop below 100%. Which one constitutes an improvement? It depends, but for the airline utilization at 100% is already what they want (by the numbers, should be their maximum profit) so it can't be improved. Only capacity can be improved.
- matchagaucho 4y agoas you inevitably have idle workers waiting for new messages. This is no longer the case using the latest in serverless IaaS architecture, where the pool of queue workers dynamically expand to meet demand.
- orf 4y agoThis has always been a thing, far far before the “latest in IaaS architecture”. The idle polling still exists except they manage it for you. Their margin is then improved if they intelligently reduce wasted polls through a variety of techniques. However, yes, you don’t pay for this or care about it much. It’s good.
- dx034 4y agoSome server will still be idle and you'll pay for it. Dedicated hardware utilized close to 100% will be massively cheaper than IaaS.