15 ms·

Queues should be empty

119 points by redman25 4y ago

ruduhudi 4y ago
Throttling stuff is another valid use case. I‘ve used a pub/sub for sending e-mail for a newsletter because we couldn‘t send too many of them at once so the consumer would run in a fixed interval and always pick up a fixed number of messages.
kbrannigan 4y ago
I just realized how useful queues can be. Instead of letting the system fail under heavy load. You put the tasks in a holding place. They will be processed later. Computers have spoiled us with instant and now.... But even a 1 hour processing time is amazing if that task would take 3 days manually.
- yurishimo 4y ago
  There are entire languages built upon this model! Anything using the BEAM is a great example. Erlang, Elixir, Gleam, etc. The runtime acts as an event loop and if a specific program is hogging too much time on the CPU, it will get paused and thrown to the back of the execution stack. This is all transparent (relatively) for the developer. Here’s a video if you want to see an example: https://youtu.be/JvBT4XBdoUE https://youtu.be/JvBT4XBdoUE
- AstixAndBelix 4y ago
  I had a course about queue theory in my SWE degree. I left the course with a deep sense of dread because I knew most people studying CS and SWE in other universities would be oblivious to such a fundamental piece of knowledge. Basically everything that processes data has a queue attached to it, from your router to your scheduler. Not knowing anything about queue theory is such an insult to the profession of SWE. It's like being a Mech engineer and not knowing about control of dynamic systems.
  - OJFord 4y ago
    Why did you think you knew that? I'm also one of your supposed few to have studied it, I never thought that. The dread should have been that most companies/teams won't care, won't engineer on that level, will just set it up however and make it bigger/faster/more-consumed if it's too slow.
    - setr 4y ago
      In my uni, the class was more or less unadvertised and the professor largely ignored — you had to go out of your way to take it (I was actively looking for poorly rated professors, because I figured they were usually rated such because they graded hard… which probably meant they cared more). When I tried to google the subject/code examples afterwards, I largely only found textbooks and one-off blog posts. Also a lot of GitHub repos with names like CS550… Only place I’ve seen it referenced since that class was: 1. HN periodically 2. A YouTube documentary about Disneyland fast pass queues 3. A business optimization consultancy Ultimately, a pretty niche subject area AFAICT. But implementing a fairly efficient queuing simulation seems to be pretty easy, and the subject seems to be stupid powerful and broadly applicable… so I’ve never understood why it’s not more discussed
      - OJFord 4y ago
        Compared to what, as if say information theory comes up (explicitly) all the time? Seems like you could say that about any more theoretically oriented course.
- roflyear 4y ago
  Except most times people don't implement the surrounding infra correctly. Let's say you consume a job, do something, but that something fails. You have to set it to be rescheduled somewhere. Lets say your things writing jobs to the queue fails. That needs to be handled as well. "They will be processed later [if I use a queue]" is not true by default (without work and thought) and assuming so will get you in trouble, and will also make certain people on your team who do understand that isn't the case upset.
Akronymus 4y ago
Queues also enable "easy" batching of jobs. (For a very limited set of problems)
orf 4y ago
No they shouldn’t. The way I see it is that there are two opposing things you can optimise for that depend on queue depth: utilisation or latency. If you care about processing each message as quickly as possible then queues should be empty. This often requires a higher cost and lower utilisation as you inevitably have idle workers waiting for new messages. If you care about utilisation, then you never want your queue to be empty while something is polling it. This might be some background task that runs on a GPU - every second that sits idle is wasted cash, and those tasks usually benefit heavily from batching inputs together. In this case you want it to always have something to read from the queue and shut it down the moment this isn’t the case.
- deleted 4y ago
  [deleted]
- deleted 4y ago
  [deleted]
- anonymoushn 4y ago
  Seems like there's some implicit assumption in this comment that everything is rented by the minute and latency never matters. Renting things by the minute isn't really the cheapest way to buy lots of things though. Also, its sometimes nice to respond to the content of the article instead of its title.
  - orf 4y ago
    I think the implicit assumption in the comment is that it would be possible for readers to understand the underlying idea of the comment rather than only see the specific example. “resource cost per minute” is not the only thing helped by a non-empty queue. Also please don’t accuse someone of not “responding to the content of the article” when the content is a bog standard list. If you want my response to that: “yes, that is indeed a list of 4 things queues help with”. Not sure it adds much to the discussion.
    - anonymoushn 4y ago
edejong 4y ago
Queues should absolutely not be empty. If they are often empty, you're over provisioning your consumers. There is an ideal size to a queue, based on the balance equation (https://en.wikipedia.org/wiki/Balance_equation https://en.wikipedia.org/wiki/Balance_equation). Also study queueing theory if you want to know more: https://en.wikipedia.org/wiki/Queueing_theory https://en.wikipedia.org/wiki/Queueing_theory.
- clnq 4y ago
  They should be empty if you wish to minimize latency. For example, when emergency room queues are full in hospitals, that is a very undesirable scenario. The same can be applied in software - there are parts of systems that should only use queues to handle spike workloads. And the longer these queues are on average, the worse the overall performance of the system is. Just like patient health outcomes drop with longer ER queues. And really, we need more context to say whether a queue should be empty, trend towards being empty, or towards always having tasks. I don't think it's possible to make a general prescription for all queues.
  - edejong 4y ago
    What are spike workloads? Is this poisson? What’s the distribution?
    - clnq 4y ago
      Distributions are useful for optimizing. But the decision whether a queue should generally be empty or have elements comes out of system requirements. Then you can employ statistics to plan what resources you will allocate to processing the queue. Though having capacity for Poisson event spikes isn’t nearly the only way to ensure queues trend to zero elements. For example, you can also scale the workload for each queue element depending on the length of the queue. It’s very popular in video game graphics to allocate a given amount of frame time for ray tracing or anti aliasing, and temporally accumulate the results. So we may process a frame in a real time rendering pipeline for more or less time, depending on the state of the pipeline. Say, if the GPU is still drawing the previous frame, we can accumulate some ray traced lighting info on the CPU for the next frame for longer. If the CPU can’t spit out frames fast enough, we might accumulate ray traced lighting kernels/points for less time each frame and scale their size/impact. Statistical distributions of frame timings don’t figure here much as workloads are unpredictable and may spike at any time - you have to adjust/scale rendering quality on the fly to maintain stable frame rates. This example assumes that we’re preparing one frame on the CPU while drawing the previous one on the GPU. But there may also be other threads that handle frames in parallel. Other workload spikes could be predicted by things like Poisson distributions. But once again, the key takeaway is that you can’t generalise this stuff in any practical application. It all depends on system requirements and context.
philliphaydon 4y ago
I think the title is meant to be: “Error queues should be empty”
smadge 4y ago
The most important reason for queues is the variability of job inter-arrival times and service times. If there was no variability there would be no need for queuing.
edejong 4y ago
Queues should have an ideal size based on your latency requirements. Say, you have 100 requests coming in per second. Each consumer of your queue requires 0.3 seconds per request. Now, you can optimise for the number of consumers. Would you choose 40 consumers, then the probability of serving the request with an empty queue in 0.3 seconds would be 17% (and > 0.3 seconds would be 84%). However, the total waiting time per request would be 0.1 second. With 80 consumers, the probability of an empty queue would be higher at 58%. However, most of those consumers are doing nothing, while your waiting time is only reduced by 0.1 second! Only a 30% improvement. (edit: probability of empty queue in second example is higher, not lower)
- anonymoushn 4y ago
  Can you state the rest of your assumptions? What is the distribution of the request arrival times?
  - edejong 4y ago
    Poisson. And 1.0 variance in request processing time.
- ThrustVectoring 4y ago
  I'm not awake/invested enough to make up numbers and do the math here, but shouldn't the cost for provisioning workers and value provided at various latencies matter? Like, each worker you provision costs a certain amount and shifts the latency distribution towards zero by a certain amount, and in principle there should be some function that converts each distribution to a dollar value. Like, suppose you had a specific service level agreement that requires X% of requests to be served within Y seconds over some time interval, and breaking the SLA costs Z dollars. Each provisioning level would generate some distribution of latencies, which can get turned into likelihood of meeting the SLA, and from there you can put a dollar value on it. And crucially, this allows the "ideal" amount of provisioning to vary based off the relative cost of over and under-provisioning; if workers are cheap and breaking the SLA is costly, you would have more workers than if the SLA is relatively unimportant and workers are expensive.
unxdfa 4y ago
I think this could be better described as "queues should be heading in the direction of empty"
- remram 4y ago
  If you look at a queue at any random time, it will have necessarily increased more than it has decreased over its lifetime, otherwise it would hold negative items.
  - unxdfa 4y ago
    That’s why you look at the average age of the items in the queue.
    - remram 4y ago
      That's a good measure (average latency) but that's very different from your earlier statement of "heading in the direction of empty".
      - unxdfa 4y ago
        Well not really. If the average age is increasing it’s not heading in the direction of empty.
        remram 4y ago
        You still haven't defined what "heading in the direction of empty" means. Like I said, your queue can't be decreasing in size over its history, and it cannot be decreasing in size most of the time either, if it started empty.
mtlmtlmtlmtl 4y ago
If a datastructure is usually empty, you probably don't need that datastructure.
- deleted 4y ago
  [deleted]
wielebny 4y ago
Also, water should be wet and SQL tables need to have primary keys.
jojobas 4y ago
Is this an attempt at reinventing queue theory? In the US it was called queue theory, in the USSR it was known as mass service theory. The thing is, there were no queues in the US and no mass service in the USSR.
denton-scratch 4y ago
The ideal queue length is one. That will minimize latency, and no poll will result in a nothingburger.
- roflyear 4y ago
  But this is not true. "Always having something to process" is not the ideal length. It will cause major problems, though it is somewhat counterintuitive. See: https://blog.danslimmon.com/2016/08/26/the-most-important-thing-to-understand-about-queues/ https://blog.danslimmon.com/2016/08/26/the-most-important-th...
  - denton-scratch 4y ago
    I didn't say the ideal was "always having something to process"; I said the ideal was always having exactly one thing to process. Come on, it's an ideal; it doesn't have to be realistic. [Edit] Just read the Slimmons article. His argument is specifically about a queue served by a server that is under stress. His assertion is: "As you approach maximum throughput, average queue size – and therefore average wait time – approaches infinity." He's quite right (and yes, it's a bit counter-intuitive). But it's orthogonal to the matter of the ideal queue length. Thanks for the link.
    - roflyear 4y ago
      Right but in reality you can't actually hit "always having exactly one thing to process" without maxing out your queue. It's actually impossible! Because they are actually the same thing. "Always having something to process" means no idle time - which means your queue is forever long :) Aiming for that ideal is not a good idea because it's misguided (for lack of a better term).
fwlr 4y ago
Spiky workloads, safely persisting data, decoupling components, and enabling asynchronous processing are all things that can be provided by a stack or a heap as well. The call stack in any given programming language is a good example of a stack for handling spiky workloads, for instance. A min-heap is more flexible than a queue for handling async/decoupled/pubsub stuff (and arguably more correct for many use cases). The thing that queues are good for is enforcing order: when I get to the front of the queue and it’s time to do my thing, I have a guarantee that anyone who entered the queue before me has already done their thing, and anyone who entered the queue after me has not yet had a chance to do their thing. This is a useful guarantee for tasks that have an order dependence, for example database operations looking to guarantee consistency.
HarHarVeryFunny 4y ago
It depends on what the queue is being used for. If a queue is just being used to decouple a producer/consumer pair, then "it should be [almost] empty" is a reasonable notion, assuming consumer/producer are compute bound and there's no fundamental reason why one should be faster than the other... Which brings us to the other type of use where things are being enqueued specifically as a means of waiting for resources that are in limited supply: maybe just compute if these are computationally heavy jobs, or maybe I/O bandwidth, etc. In that case we expect the queue to often be non-empty, and there's nothing wrong with that, even if there's potential to throw money/hardware at it to get the queue size down to zero!
nkozyra 4y ago
The goal of a queue is to be empty, the purpose of a queue is to occasionally not be.
- robotresearcher 4y ago
  Unless the goal of your queue is to avoid starving consumers. Then it should always be non-empty.
MaxGabriel 4y ago
Another benefit, queues can enforce ordering, and limit concurrency. This can be helpful for avoiding resource contention, or hitting an API limit.
- remram 4y ago
  Queue can also allow batching. It might be that you need to consume or pack or ship N items at a time, otherwise you're losing money. Trying to aim for minimal latency would literally hurt your business. This article obviously has a very specific kind of workload/service/target in mind that they fail to define and they overgeneralize. This is bad advice in many situations that they failed to imagine.
kilgnad 4y ago
He's right. You guys need to see the system in terms of it's derivative. The rate of speed in which data is produced and consumed. This article is saying you basically want an average negative velocity where consumers eat data faster then producers on average. Any queue that is non empty means the velocity is positive. Producers are making data faster then consumers and you have a limited time window before the queue becomes full and your system starts dropping data. If you observe that your system never has a full queue but is constantly loaded with partially filled queues it means you built and tuned your system such that it operates on the border of failure. It means your system is continuously hitting unsustainable velocities all the time, but stops short of total failure (a full queue). Remember, think about the system in terms of it's derivative. The average derivative must be negative for your system to even function. Any time a queue is filled partially or full it means there was a period of positive velocity. These periods cannot exceed the amount of time the system is in negative velocity.
bob1029 4y ago
I have no strong opinion on this as long as you are not using queues for purposes of inter-thread communication. If you are using queues for this purpose, then I recommend reading the following: https://lmax-exchange.github.io/disruptor/disruptor.html#_the_problems_of_queues https://lmax-exchange.github.io/disruptor/disruptor.html#_th... > When in use, queues are typically always close to full or close to empty due to the differences in pace between consumers and producers. They very rarely operate in a balanced middle ground where the rate of production and consumption is evenly matched. This has always been my experience. Has anyone ever had a perfect goldilocks scenario between 2 threads?
- wongarsu 4y ago
  One good use-case for still using queues is if your producer or consumer has spiky characteristics on small timescales, but predicatable perfomance if you zoom out enough. In that case your queue size spikes to some high value, but always returns to 0. The reasons for this can be numerous, maybe you produce a lot of jobs at once because of a user interaction, or your consumer is running slower for five minutes because some cron job is stealing your resources.
ysavir 4y ago
The author has a good sentiment, but wrong expression. It's not that "queues should be empty", it's that "you should be emptying your queues". Whether they reach an empty state is irrelevant, it's just a sisyphian task that should always be working towards empty, and ideally, never actually reaching empty. A good comparison is a restaurant's order queue. If the order queue is always empty, that means no one is ordering food at the restaurant. That is an undesirable state. We want people to be ordering food, an empty queue shows that we have a problem in getting people to eat at our restaurant. We don't want our queue to be always growing. If it's always growing, that means we're understaffed, and people waiting for their meals will get frustrated. Some will wait and leave a bad review, others might leave before they ever get their meal--and won't pay for it, either. Lost time and effort on our part. But an order queue that almost always has two meals in the making is golden. It means we're getting consistent clientele to dine at our restaurant, that meals are being cooked and served in good time, diners are happy with the experience, and our system operations are healthy. I think the author of TFA understands this, but stopped a little short of the optimal expression of their ideas. It's not about what you have, it's about what you should always be working towards.
- colechristensen 4y ago
  You’ve got it all wrong. If you’re running a restaurant and orders never back up then you have too much staff, too big a kitchen, and are losing money by having too much capacity. You want to have a lunch rush, for some people to leave because you’re too busy. When the queue is backed up is when you’re the most profitable both per hour and per unit sold. Then you want to find efficiencies to improve throughout at the times you’re backed up. You need to intentionally design your bottlenecks not pretend that you can avoid them completely.
  - ysavir 4y ago
    I considered addressing that, but decided that while accurate, it was getting nitpicky in the context of the article.
highspeedbus 4y ago
>Decoupling components >This way, the producer doesn’t need to know anything about the consumer and vice versa. Both components needs to agree on shared data, down to what each field actually means. The producer is just not saying What To Do with the message. Take an example of processing a purchase order in multiple, asynchronous steps. While the webpage button spins there's a whole dance of agreed messages and expected outcomes that needs to happen in order for it to work. And there is no decoupling in sight. My take is that Decoupling is a vague word. One should design distributed systems assuming that coupling is manageable, not inexistent. >you could submit the task and ACK immediately allowing the job to finish in the background. That's bad practice.
plandis 4y ago
> If the queue is not empty, it means that whatever job is consuming from the queue is not keeping up with messages coming into it. You could conceivably have a non-zero queue size that asymptotically reaches some stable size over time. In such a case you are keeping up with messages but with some approximately static amount of added latency. I don’t think I’ve seen this happen in practice but I think some systems can be engineered to be approximately stable (grows and shrinks but within predictable bounds)
- remram 4y ago
  This is very common when you can adjust the consume rate. This happens in practice: serverless platforms scale, Kubernetes has HorizontalPodAutoscaler, you can manually grab more VMs, or IRL this might mean calling in more staff or at larger time scales hiring more people in your organization.
tweaktastic 4y ago
I agree with author on the point that queues are best to handle spikes. But when you are talking about scale dealing with millions of messages in short span of time for extended period, queues can never remain empty. Empty queues in this situation would mean that your concurrency is high. When your concurrency is high, this would result in too many concurrent connections to your database which wouldend upcreating bottlenecks for your system. The system should be designed to handle as many messages as possible ensuring that your resources don't get over burdened. Empty queues where your messages keep getting rejected because your resources are being hammered is not necessarily an efficient system.
- roflyear 4y ago
  > When your concurrency is high, this would result in too many concurrent connections to your database which wouldend upcreating bottlenecks for your system. Can you expand on this?
  - dmoy 4y ago
    If you have a polling queue, as the number of workers grows too large, the overhead of maintaining connections from (worker) -> (stuff in the middle) -> (db or whatever keeps queue state) becomes too high. In the extreme case, you can get into a point of diminishing returns, where adding more workers hurts both throughput and latency in a bad way. That is - initially you can decrease latency by sacrificing utilization and just adding more workers that usually sit idle. This increases throughput during heavy load, and decreases latency. Until you can't, because the overhead from all the extra workers causes a slowdown up the stack (either the controller that's handling messages, or the db itself that's got the queue state, or whatever is up there). Then when you throw more overhead, you increase latency and decrease throughput, because of the extra time wasted on overhead. It depends on a lot of different variables. If you have work items that vary significantly in cost, it can add even more problems (i.e. if your queue items can vary 3+ orders of magnitude in processing cost).
    - roflyear 4y ago
gillh 4y ago
HN: our team has been working on an advanced API queue/load management system called Aperture[0]. It has following capabilities - 1. Limits concurrency based on latency gradient or queue depths etc. 2. Weighted fair queueing scheduler - prioritized load shedding based on workload priorities and latency estimation. Anyone interested in Little’s Law, token buckets, network schedulers, control theory (PID controllers) should definitely check it out! [1] [0]: https://github.com/fluxninja/aperture https://github.com/fluxninja/aperture [1]: https://docs.fluxninja.com/ https://docs.fluxninja.com/
dylan604 4y ago
Should this title be given some sort of limiter/qualifier as in "Messaging queues should be empty"? I come from the video production world where you have things like render queues. These are designed to utilize a limited number (due to sheer expense) of hardware instances where if they finished the jobs so fast that the queue was always empty would indicate you have too many very expensive machines. A queue that is never empty, but is just finishing as new big pushes to the queue happen would be the ideal balance.
taeric 4y ago
Alternatively stated, queues are safest in a steady state. If they are growing in size, that is clearly bad if not bounded. If they are shrinking in size, that means you are going to be idling soon. I think it is fair that if you are picking between those states, you'd prefer the idling. That said, consider a physical machine. The "hopper" on many is essentially a queue to feed in items to work on. If you let that go empty, you messed up. Hard.
8note 4y ago
An always empty queue means you could run into problems when traffic increases. I've seen this come up recently, where our queue reader was(well, still is) over scaled, where it could handle 100x the normal queue throughout. When the input rate gets higher though, instead of the queue filling a bit and controlling the traffic, we hammer our dependencies. I now see a continuously empty queue as a smell. It should be filling and emptying over and over again. Not all the time, but a queue that never fills is a ticking time bomb
- advisedwang 4y ago
  What you're describing (empty queue = unprepared for load) is a code smell, but the solution doesn't have to be "fill the queue sometimes". It could also be: * (Mandatory) Have a back-pressure system. * (Optional, depends on system requirements) have auto-scaling of queue readers. * (Mandatory) Make sure the system is regularly load tested.
c3534l 4y ago
This is a somewhat uninsightful observation that only has people responding to it because of pedantic arguing. I don't think anyone even disagrees with the author, they're just arguing about words.
ww520 4y ago
Queues are great as a buffer to even out compute utilization. With an empty queue, most workers can be shut down after a while, saving on compute cost. When the queue grows and stays non-empty, more workers can be started up. The length of the queue can be used to estimate the number of workers to spin up, as a metric for real time capacity planning.
Taniwha 4y ago
Think about the medical system, people hate to be put onto waiting lists (queues), but empty queues means that extremely expensive resources (operating theatres, staff, surgeons) are sitting idle which is very inefficient. Ideally queues mean that another patient is always there when the operating theatre becomes idle. Because medical need is not smooth and constant sometimes they fill up, sometimes they drain - they allow planners to match average need to resources. So non-empty queues are a good thing, even for hospital waiting lists. (To forestall arguments, of course the operating theatre example is not a pure queue example, some patients get put to the head of the queue according to triage - for the purpose of this example it doesn't matter)
mindentropy 4y ago
Queues can also be used to coalesce events. You can run through the queue and see if there are repetitive events which does not have any effect multiple times and can be combined to a single event. This would reduce the work done at the consumer processing these events.
vivegi 4y ago
In a complex workflow that has queues acting as buffers for work items between processes, the intermediate queue sizes depends on the turnaround time promised to the end-customer (which is another way of saying rate of customer demand). If overall customer demand is x widgets per unit of time, if you design your system to process y widgets per unit of time, you will have overprovisioning (and hence impact to profits) if y >> x (y very large relative to x). As a corollary, if you design your system where y < x (y smaller than x), then you will not meet SLAs for some customers. This analysis can be done at each subsystem level that is a component workflow and in reality some subsystems may be overprovisioned while some are underprovisioned. The goal should be to match the overall rate of customer demand. You also design the system to handle a certain level of spike in demand, but the system will fail when demand exceeds capacity. When that happens, you want one place where you want to manage flow and invariably that will be a critical queue.

119 points by redman25 4y ago

Throttling stuff is another valid use case. I‘ve used a pub/sub for sending e-mail for a newsletter because we couldn‘t send too many of them at once so the consumer would run in a fixed interval and always pick up a fixed number of messages.

kbrannigan 4y ago

I just realized how useful queues can be. Instead of letting the system fail under heavy load. You put the tasks in a holding place. They will be processed later. Computers have spoiled us with instant and now.... But even a 1 hour processing time is amazing if that task would take 3 days manually.

yurishimo 4y ago
There are entire languages built upon this model! Anything using the BEAM is a great example. Erlang, Elixir, Gleam, etc. The runtime acts as an event loop and if a specific program is hogging too much time on the CPU, it will get paused and thrown to the back of the execution stack. This is all transparent (relatively) for the developer. Here’s a video if you want to see an example: https://youtu.be/JvBT4XBdoUE https://youtu.be/JvBT4XBdoUE
AstixAndBelix 4y ago
I had a course about queue theory in my SWE degree. I left the course with a deep sense of dread because I knew most people studying CS and SWE in other universities would be oblivious to such a fundamental piece of knowledge. Basically everything that processes data has a queue attached to it, from your router to your scheduler. Not knowing anything about queue theory is such an insult to the profession of SWE. It's like being a Mech engineer and not knowing about control of dynamic systems.
- OJFord 4y ago
  Why did you think you knew that? I'm also one of your supposed few to have studied it, I never thought that. The dread should have been that most companies/teams won't care, won't engineer on that level, will just set it up however and make it bigger/faster/more-consumed if it's too slow.
  - setr 4y ago
    In my uni, the class was more or less unadvertised and the professor largely ignored — you had to go out of your way to take it (I was actively looking for poorly rated professors, because I figured they were usually rated such because they graded hard… which probably meant they cared more). When I tried to google the subject/code examples afterwards, I largely only found textbooks and one-off blog posts. Also a lot of GitHub repos with names like CS550… Only place I’ve seen it referenced since that class was: 1. HN periodically 2. A YouTube documentary about Disneyland fast pass queues 3. A business optimization consultancy Ultimately, a pretty niche subject area AFAICT. But implementing a fairly efficient queuing simulation seems to be pretty easy, and the subject seems to be stupid powerful and broadly applicable… so I’ve never understood why it’s not more discussed
    - OJFord 4y ago
      Compared to what, as if say information theory comes up (explicitly) all the time? Seems like you could say that about any more theoretically oriented course.
roflyear 4y ago
Except most times people don't implement the surrounding infra correctly. Let's say you consume a job, do something, but that something fails. You have to set it to be rescheduled somewhere. Lets say your things writing jobs to the queue fails. That needs to be handled as well. "They will be processed later [if I use a queue]" is not true by default (without work and thought) and assuming so will get you in trouble, and will also make certain people on your team who do understand that isn't the case upset.

Akronymus 4y ago

Queues also enable "easy" batching of jobs. (For a very limited set of problems)

orf 4y ago

No they shouldn’t. The way I see it is that there are two opposing things you can optimise for that depend on queue depth: utilisation or latency. If you care about processing each message as quickly as possible then queues should be empty. This often requires a higher cost and lower utilisation as you inevitably have idle workers waiting for new messages. If you care about utilisation, then you never want your queue to be empty while something is polling it. This might be some background task that runs on a GPU - every second that sits idle is wasted cash, and those tasks usually benefit heavily from batching inputs together. In this case you want it to always have something to read from the queue and shut it down the moment this isn’t the case.

deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
anonymoushn 4y ago
Seems like there's some implicit assumption in this comment that everything is rented by the minute and latency never matters. Renting things by the minute isn't really the cheapest way to buy lots of things though. Also, its sometimes nice to respond to the content of the article instead of its title.
- orf 4y ago
  I think the implicit assumption in the comment is that it would be possible for readers to understand the underlying idea of the comment rather than only see the specific example. “resource cost per minute” is not the only thing helped by a non-empty queue. Also please don’t accuse someone of not “responding to the content of the article” when the content is a bog standard list. If you want my response to that: “yes, that is indeed a list of 4 things queues help with”. Not sure it adds much to the discussion.
  - anonymoushn 4y ago

edejong 4y ago

Queues should absolutely not be empty. If they are often empty, you're over provisioning your consumers. There is an ideal size to a queue, based on the balance equation (https://en.wikipedia.org/wiki/Balance_equation https://en.wikipedia.org/wiki/Balance_equation). Also study queueing theory if you want to know more: https://en.wikipedia.org/wiki/Queueing_theory https://en.wikipedia.org/wiki/Queueing_theory.

clnq 4y ago
They should be empty if you wish to minimize latency. For example, when emergency room queues are full in hospitals, that is a very undesirable scenario. The same can be applied in software - there are parts of systems that should only use queues to handle spike workloads. And the longer these queues are on average, the worse the overall performance of the system is. Just like patient health outcomes drop with longer ER queues. And really, we need more context to say whether a queue should be empty, trend towards being empty, or towards always having tasks. I don't think it's possible to make a general prescription for all queues.
- edejong 4y ago
  What are spike workloads? Is this poisson? What’s the distribution?
  - clnq 4y ago
    Distributions are useful for optimizing. But the decision whether a queue should generally be empty or have elements comes out of system requirements. Then you can employ statistics to plan what resources you will allocate to processing the queue. Though having capacity for Poisson event spikes isn’t nearly the only way to ensure queues trend to zero elements. For example, you can also scale the workload for each queue element depending on the length of the queue. It’s very popular in video game graphics to allocate a given amount of frame time for ray tracing or anti aliasing, and temporally accumulate the results. So we may process a frame in a real time rendering pipeline for more or less time, depending on the state of the pipeline. Say, if the GPU is still drawing the previous frame, we can accumulate some ray traced lighting info on the CPU for the next frame for longer. If the CPU can’t spit out frames fast enough, we might accumulate ray traced lighting kernels/points for less time each frame and scale their size/impact. Statistical distributions of frame timings don’t figure here much as workloads are unpredictable and may spike at any time - you have to adjust/scale rendering quality on the fly to maintain stable frame rates. This example assumes that we’re preparing one frame on the CPU while drawing the previous one on the GPU. But there may also be other threads that handle frames in parallel. Other workload spikes could be predicted by things like Poisson distributions. But once again, the key takeaway is that you can’t generalise this stuff in any practical application. It all depends on system requirements and context.

philliphaydon 4y ago

I think the title is meant to be: “Error queues should be empty”

smadge 4y ago

The most important reason for queues is the variability of job inter-arrival times and service times. If there was no variability there would be no need for queuing.

edejong 4y ago

Queues should have an ideal size based on your latency requirements. Say, you have 100 requests coming in per second. Each consumer of your queue requires 0.3 seconds per request. Now, you can optimise for the number of consumers. Would you choose 40 consumers, then the probability of serving the request with an empty queue in 0.3 seconds would be 17% (and > 0.3 seconds would be 84%). However, the total waiting time per request would be 0.1 second. With 80 consumers, the probability of an empty queue would be higher at 58%. However, most of those consumers are doing nothing, while your waiting time is only reduced by 0.1 second! Only a 30% improvement. (edit: probability of empty queue in second example is higher, not lower)

anonymoushn 4y ago
Can you state the rest of your assumptions? What is the distribution of the request arrival times?
- edejong 4y ago
  Poisson. And 1.0 variance in request processing time.
ThrustVectoring 4y ago
I'm not awake/invested enough to make up numbers and do the math here, but shouldn't the cost for provisioning workers and value provided at various latencies matter? Like, each worker you provision costs a certain amount and shifts the latency distribution towards zero by a certain amount, and in principle there should be some function that converts each distribution to a dollar value. Like, suppose you had a specific service level agreement that requires X% of requests to be served within Y seconds over some time interval, and breaking the SLA costs Z dollars. Each provisioning level would generate some distribution of latencies, which can get turned into likelihood of meeting the SLA, and from there you can put a dollar value on it. And crucially, this allows the "ideal" amount of provisioning to vary based off the relative cost of over and under-provisioning; if workers are cheap and breaking the SLA is costly, you would have more workers than if the SLA is relatively unimportant and workers are expensive.

unxdfa 4y ago

I think this could be better described as "queues should be heading in the direction of empty"

remram 4y ago
If you look at a queue at any random time, it will have necessarily increased more than it has decreased over its lifetime, otherwise it would hold negative items.
- unxdfa 4y ago
  That’s why you look at the average age of the items in the queue.
  - remram 4y ago
    That's a good measure (average latency) but that's very different from your earlier statement of "heading in the direction of empty".
    - unxdfa 4y ago
      Well not really. If the average age is increasing it’s not heading in the direction of empty.
      - remram 4y ago
        You still haven't defined what "heading in the direction of empty" means. Like I said, your queue can't be decreasing in size over its history, and it cannot be decreasing in size most of the time either, if it started empty.

mtlmtlmtlmtl 4y ago

If a datastructure is usually empty, you probably don't need that datastructure.

deleted 4y ago
[deleted]

wielebny 4y ago

Also, water should be wet and SQL tables need to have primary keys.

jojobas 4y ago

Is this an attempt at reinventing queue theory? In the US it was called queue theory, in the USSR it was known as mass service theory. The thing is, there were no queues in the US and no mass service in the USSR.

denton-scratch 4y ago

The ideal queue length is one. That will minimize latency, and no poll will result in a nothingburger.

roflyear 4y ago
But this is not true. "Always having something to process" is not the ideal length. It will cause major problems, though it is somewhat counterintuitive. See: https://blog.danslimmon.com/2016/08/26/the-most-important-thing-to-understand-about-queues/ https://blog.danslimmon.com/2016/08/26/the-most-important-th...
- denton-scratch 4y ago
  I didn't say the ideal was "always having something to process"; I said the ideal was always having exactly one thing to process. Come on, it's an ideal; it doesn't have to be realistic. [Edit] Just read the Slimmons article. His argument is specifically about a queue served by a server that is under stress. His assertion is: "As you approach maximum throughput, average queue size – and therefore average wait time – approaches infinity." He's quite right (and yes, it's a bit counter-intuitive). But it's orthogonal to the matter of the ideal queue length. Thanks for the link.
  - roflyear 4y ago
    Right but in reality you can't actually hit "always having exactly one thing to process" without maxing out your queue. It's actually impossible! Because they are actually the same thing. "Always having something to process" means no idle time - which means your queue is forever long :) Aiming for that ideal is not a good idea because it's misguided (for lack of a better term).

fwlr 4y ago

Spiky workloads, safely persisting data, decoupling components, and enabling asynchronous processing are all things that can be provided by a stack or a heap as well. The call stack in any given programming language is a good example of a stack for handling spiky workloads, for instance. A min-heap is more flexible than a queue for handling async/decoupled/pubsub stuff (and arguably more correct for many use cases). The thing that queues are good for is enforcing order: when I get to the front of the queue and it’s time to do my thing, I have a guarantee that anyone who entered the queue before me has already done their thing, and anyone who entered the queue after me has not yet had a chance to do their thing. This is a useful guarantee for tasks that have an order dependence, for example database operations looking to guarantee consistency.

HarHarVeryFunny 4y ago

It depends on what the queue is being used for. If a queue is just being used to decouple a producer/consumer pair, then "it should be [almost] empty" is a reasonable notion, assuming consumer/producer are compute bound and there's no fundamental reason why one should be faster than the other... Which brings us to the other type of use where things are being enqueued specifically as a means of waiting for resources that are in limited supply: maybe just compute if these are computationally heavy jobs, or maybe I/O bandwidth, etc. In that case we expect the queue to often be non-empty, and there's nothing wrong with that, even if there's potential to throw money/hardware at it to get the queue size down to zero!

nkozyra 4y ago

The goal of a queue is to be empty, the purpose of a queue is to occasionally not be.

robotresearcher 4y ago
Unless the goal of your queue is to avoid starving consumers. Then it should always be non-empty.

MaxGabriel 4y ago

Another benefit, queues can enforce ordering, and limit concurrency. This can be helpful for avoiding resource contention, or hitting an API limit.

remram 4y ago
Queue can also allow batching. It might be that you need to consume or pack or ship N items at a time, otherwise you're losing money. Trying to aim for minimal latency would literally hurt your business. This article obviously has a very specific kind of workload/service/target in mind that they fail to define and they overgeneralize. This is bad advice in many situations that they failed to imagine.

kilgnad 4y ago

He's right. You guys need to see the system in terms of it's derivative. The rate of speed in which data is produced and consumed. This article is saying you basically want an average negative velocity where consumers eat data faster then producers on average. Any queue that is non empty means the velocity is positive. Producers are making data faster then consumers and you have a limited time window before the queue becomes full and your system starts dropping data. If you observe that your system never has a full queue but is constantly loaded with partially filled queues it means you built and tuned your system such that it operates on the border of failure. It means your system is continuously hitting unsustainable velocities all the time, but stops short of total failure (a full queue). Remember, think about the system in terms of it's derivative. The average derivative must be negative for your system to even function. Any time a queue is filled partially or full it means there was a period of positive velocity. These periods cannot exceed the amount of time the system is in negative velocity.

bob1029 4y ago

I have no strong opinion on this as long as you are not using queues for purposes of inter-thread communication. If you are using queues for this purpose, then I recommend reading the following: https://lmax-exchange.github.io/disruptor/disruptor.html#_the_problems_of_queues https://lmax-exchange.github.io/disruptor/disruptor.html#_th... > When in use, queues are typically always close to full or close to empty due to the differences in pace between consumers and producers. They very rarely operate in a balanced middle ground where the rate of production and consumption is evenly matched. This has always been my experience. Has anyone ever had a perfect goldilocks scenario between 2 threads?

wongarsu 4y ago
One good use-case for still using queues is if your producer or consumer has spiky characteristics on small timescales, but predicatable perfomance if you zoom out enough. In that case your queue size spikes to some high value, but always returns to 0. The reasons for this can be numerous, maybe you produce a lot of jobs at once because of a user interaction, or your consumer is running slower for five minutes because some cron job is stealing your resources.

ysavir 4y ago

The author has a good sentiment, but wrong expression. It's not that "queues should be empty", it's that "you should be emptying your queues". Whether they reach an empty state is irrelevant, it's just a sisyphian task that should always be working towards empty, and ideally, never actually reaching empty. A good comparison is a restaurant's order queue. If the order queue is always empty, that means no one is ordering food at the restaurant. That is an undesirable state. We want people to be ordering food, an empty queue shows that we have a problem in getting people to eat at our restaurant. We don't want our queue to be always growing. If it's always growing, that means we're understaffed, and people waiting for their meals will get frustrated. Some will wait and leave a bad review, others might leave before they ever get their meal--and won't pay for it, either. Lost time and effort on our part. But an order queue that almost always has two meals in the making is golden. It means we're getting consistent clientele to dine at our restaurant, that meals are being cooked and served in good time, diners are happy with the experience, and our system operations are healthy. I think the author of TFA understands this, but stopped a little short of the optimal expression of their ideas. It's not about what you have, it's about what you should always be working towards.

colechristensen 4y ago
You’ve got it all wrong. If you’re running a restaurant and orders never back up then you have too much staff, too big a kitchen, and are losing money by having too much capacity. You want to have a lunch rush, for some people to leave because you’re too busy. When the queue is backed up is when you’re the most profitable both per hour and per unit sold. Then you want to find efficiencies to improve throughout at the times you’re backed up. You need to intentionally design your bottlenecks not pretend that you can avoid them completely.
- ysavir 4y ago
  I considered addressing that, but decided that while accurate, it was getting nitpicky in the context of the article.

highspeedbus 4y ago

>Decoupling components >This way, the producer doesn’t need to know anything about the consumer and vice versa. Both components needs to agree on shared data, down to what each field actually means. The producer is just not saying What To Do with the message. Take an example of processing a purchase order in multiple, asynchronous steps. While the webpage button spins there's a whole dance of agreed messages and expected outcomes that needs to happen in order for it to work. And there is no decoupling in sight. My take is that Decoupling is a vague word. One should design distributed systems assuming that coupling is manageable, not inexistent. >you could submit the task and ACK immediately allowing the job to finish in the background. That's bad practice.

plandis 4y ago

> If the queue is not empty, it means that whatever job is consuming from the queue is not keeping up with messages coming into it. You could conceivably have a non-zero queue size that asymptotically reaches some stable size over time. In such a case you are keeping up with messages but with some approximately static amount of added latency. I don’t think I’ve seen this happen in practice but I think some systems can be engineered to be approximately stable (grows and shrinks but within predictable bounds)

remram 4y ago
This is very common when you can adjust the consume rate. This happens in practice: serverless platforms scale, Kubernetes has HorizontalPodAutoscaler, you can manually grab more VMs, or IRL this might mean calling in more staff or at larger time scales hiring more people in your organization.

tweaktastic 4y ago

I agree with author on the point that queues are best to handle spikes. But when you are talking about scale dealing with millions of messages in short span of time for extended period, queues can never remain empty. Empty queues in this situation would mean that your concurrency is high. When your concurrency is high, this would result in too many concurrent connections to your database which wouldend upcreating bottlenecks for your system. The system should be designed to handle as many messages as possible ensuring that your resources don't get over burdened. Empty queues where your messages keep getting rejected because your resources are being hammered is not necessarily an efficient system.

roflyear 4y ago
> When your concurrency is high, this would result in too many concurrent connections to your database which wouldend upcreating bottlenecks for your system. Can you expand on this?
- dmoy 4y ago
  If you have a polling queue, as the number of workers grows too large, the overhead of maintaining connections from (worker) -> (stuff in the middle) -> (db or whatever keeps queue state) becomes too high. In the extreme case, you can get into a point of diminishing returns, where adding more workers hurts both throughput and latency in a bad way. That is - initially you can decrease latency by sacrificing utilization and just adding more workers that usually sit idle. This increases throughput during heavy load, and decreases latency. Until you can't, because the overhead from all the extra workers causes a slowdown up the stack (either the controller that's handling messages, or the db itself that's got the queue state, or whatever is up there). Then when you throw more overhead, you increase latency and decrease throughput, because of the extra time wasted on overhead. It depends on a lot of different variables. If you have work items that vary significantly in cost, it can add even more problems (i.e. if your queue items can vary 3+ orders of magnitude in processing cost).
  - roflyear 4y ago

gillh 4y ago

HN: our team has been working on an advanced API queue/load management system called Aperture[0]. It has following capabilities - 1. Limits concurrency based on latency gradient or queue depths etc. 2. Weighted fair queueing scheduler - prioritized load shedding based on workload priorities and latency estimation. Anyone interested in Little’s Law, token buckets, network schedulers, control theory (PID controllers) should definitely check it out! [1] [0]: https://github.com/fluxninja/aperture https://github.com/fluxninja/aperture [1]: https://docs.fluxninja.com/ https://docs.fluxninja.com/

dylan604 4y ago

Should this title be given some sort of limiter/qualifier as in "Messaging queues should be empty"? I come from the video production world where you have things like render queues. These are designed to utilize a limited number (due to sheer expense) of hardware instances where if they finished the jobs so fast that the queue was always empty would indicate you have too many very expensive machines. A queue that is never empty, but is just finishing as new big pushes to the queue happen would be the ideal balance.

taeric 4y ago

Alternatively stated, queues are safest in a steady state. If they are growing in size, that is clearly bad if not bounded. If they are shrinking in size, that means you are going to be idling soon. I think it is fair that if you are picking between those states, you'd prefer the idling. That said, consider a physical machine. The "hopper" on many is essentially a queue to feed in items to work on. If you let that go empty, you messed up. Hard.

8note 4y ago

An always empty queue means you could run into problems when traffic increases. I've seen this come up recently, where our queue reader was(well, still is) over scaled, where it could handle 100x the normal queue throughout. When the input rate gets higher though, instead of the queue filling a bit and controlling the traffic, we hammer our dependencies. I now see a continuously empty queue as a smell. It should be filling and emptying over and over again. Not all the time, but a queue that never fills is a ticking time bomb

advisedwang 4y ago
What you're describing (empty queue = unprepared for load) is a code smell, but the solution doesn't have to be "fill the queue sometimes". It could also be: * (Mandatory) Have a back-pressure system. * (Optional, depends on system requirements) have auto-scaling of queue readers. * (Mandatory) Make sure the system is regularly load tested.

c3534l 4y ago

This is a somewhat uninsightful observation that only has people responding to it because of pedantic arguing. I don't think anyone even disagrees with the author, they're just arguing about words.

ww520 4y ago

Queues are great as a buffer to even out compute utilization. With an empty queue, most workers can be shut down after a while, saving on compute cost. When the queue grows and stays non-empty, more workers can be started up. The length of the queue can be used to estimate the number of workers to spin up, as a metric for real time capacity planning.

Taniwha 4y ago

Think about the medical system, people hate to be put onto waiting lists (queues), but empty queues means that extremely expensive resources (operating theatres, staff, surgeons) are sitting idle which is very inefficient. Ideally queues mean that another patient is always there when the operating theatre becomes idle. Because medical need is not smooth and constant sometimes they fill up, sometimes they drain - they allow planners to match average need to resources. So non-empty queues are a good thing, even for hospital waiting lists. (To forestall arguments, of course the operating theatre example is not a pure queue example, some patients get put to the head of the queue according to triage - for the purpose of this example it doesn't matter)

mindentropy 4y ago

Queues can also be used to coalesce events. You can run through the queue and see if there are repetitive events which does not have any effect multiple times and can be combined to a single event. This would reduce the work done at the consumer processing these events.

vivegi 4y ago

In a complex workflow that has queues acting as buffers for work items between processes, the intermediate queue sizes depends on the turnaround time promised to the end-customer (which is another way of saying rate of customer demand). If overall customer demand is x widgets per unit of time, if you design your system to process y widgets per unit of time, you will have overprovisioning (and hence impact to profits) if y >> x (y very large relative to x). As a corollary, if you design your system where y < x (y smaller than x), then you will not meet SLAs for some customers. This analysis can be done at each subsystem level that is a component workflow and in reality some subsystems may be overprovisioned while some are underprovisioned. The goal should be to match the overall rate of customer demand. You also design the system to handle a certain level of spike in demand, but the system will fail when demand exceeds capacity. When that happens, you want one place where you want to manage flow and invariably that will be a critical queue.