3 ms·

> should always be full > always have something to read from the queue In the first case, you've gone a bit far. To maximise utilisation, you want the queue t

> should always be full > always have something to read from the queue In the first case, you've gone a bit far. To maximise utilisation, you want the queue to never be empty when polled.

orf 4y ago
Indeed, and this is what I meant but worded poorly. Obviously you don’t want it to be 100% full, but you do want it to be the kind of full you get after a nice meal but you still have room for desert and a coffee.
- roflyear 4y ago
  Good reading here: https://blog.danslimmon.com/2016/08/26/the-most-important-thing-to-understand-about-queues/ https://blog.danslimmon.com/2016/08/26/the-most-important-th... You really don't want your queue more than 70-80% utilized. Once it gets to 75% you should consider that "red zone" and you need to fix it somehow. Once the queue is 80% you're really hitting into some issues. Above 80% and you will have significant problems.
  - orf 4y ago
    Thanks for this, that has helped me see something I missed. If you’re dealing with something like SQS and are receiving variable sized batches of messages, the consumers can be classed as “100% utilised” (always processing a batch of messages with no empty receives) whilst the queue would not be considered 100% utilised (the average delivered batch size is always less than the max).
    - roflyear 4y ago
      What issue are you trying to solve? The expense of polling? Obviously there is a line where you do not want a bunch of expensive consumers hanging around - but this isn't the place IMO to micro-optimize your costs. You shouldn't have like 20x the consumers you need but you shouldn't worry about having EXACTLY the correct amount. In your example, are you implying that if more jobs come in more workers get spawned? Is that specific to the Amazon service?
      - orf 4y ago
        If a process is busy processing a task, it is busy. A process could receive N tasks in a batch and process them at the same time. In this scenario it is possible for all workers to be utilised whilst the queue is not “100% utilised” - which is a failure mode the link you shared explained: > And by induction, we can show that a system that’s forever at 100% utilization will exceed any finite queue size you care to pick The inflows to the queue are less than the outflows, despite every worker spending 100% of its time processing tasks. It is specific on the utilisation type though, I’m mainly thinking about ML inference that has a fairly fixed initial cost per inference but scales quite nicely with >1 input.
        roflyear 4y ago
        You can't have 100% utilization of any processing capacity (ignore queues) without reaching this type of failure. 100% CPU utilization is essentially just meaning you filled up the queue on your worker server. > The inflows to the queue are less than the outflows, despite every worker spending 100% of its time processing tasks. Right, but at some point they will have to have SOME idle time. If they are NEVER idle the queue by definition is growing, not shrinking, as you said. Unless you have some system that monitors queue length and doesn't add jobs if it is like >n, which is what the article suggests (maximum queue size). But then you have to figure out what you do with those jobs that can't be added, which probably means just creating some kind of queue higher up in the pipe. If they are processing more than is being added, eventually things will run out of the queue.
  - AceyMan 4y ago
    that's a great paper that clearly explains how "back pressure" [my term] at high utilization causes latency to go hockey-stick. (And thanks for the link bc I apparently forgot to bookmark it when I initially read it.)
    - roflyear 4y ago
      You're welcome! Important for people to get this I think bc we're using this tech more and more.

> should always be full > always have something to read from the queue In the first case, you've gone a bit far. To maximise utilisation, you want the queue to never be empty when polled.

orf 4y ago

Indeed, and this is what I meant but worded poorly. Obviously you don’t want it to be 100% full, but you do want it to be the kind of full you get after a nice meal but you still have room for desert and a coffee.

roflyear 4y ago
Good reading here: https://blog.danslimmon.com/2016/08/26/the-most-important-thing-to-understand-about-queues/ https://blog.danslimmon.com/2016/08/26/the-most-important-th... You really don't want your queue more than 70-80% utilized. Once it gets to 75% you should consider that "red zone" and you need to fix it somehow. Once the queue is 80% you're really hitting into some issues. Above 80% and you will have significant problems.
- orf 4y ago
  Thanks for this, that has helped me see something I missed. If you’re dealing with something like SQS and are receiving variable sized batches of messages, the consumers can be classed as “100% utilised” (always processing a batch of messages with no empty receives) whilst the queue would not be considered 100% utilised (the average delivered batch size is always less than the max).
  - roflyear 4y ago
    What issue are you trying to solve? The expense of polling? Obviously there is a line where you do not want a bunch of expensive consumers hanging around - but this isn't the place IMO to micro-optimize your costs. You shouldn't have like 20x the consumers you need but you shouldn't worry about having EXACTLY the correct amount. In your example, are you implying that if more jobs come in more workers get spawned? Is that specific to the Amazon service?
    - orf 4y ago
      If a process is busy processing a task, it is busy. A process could receive N tasks in a batch and process them at the same time. In this scenario it is possible for all workers to be utilised whilst the queue is not “100% utilised” - which is a failure mode the link you shared explained: > And by induction, we can show that a system that’s forever at 100% utilization will exceed any finite queue size you care to pick The inflows to the queue are less than the outflows, despite every worker spending 100% of its time processing tasks. It is specific on the utilisation type though, I’m mainly thinking about ML inference that has a fairly fixed initial cost per inference but scales quite nicely with >1 input.
      - roflyear 4y ago
        You can't have 100% utilization of any processing capacity (ignore queues) without reaching this type of failure. 100% CPU utilization is essentially just meaning you filled up the queue on your worker server. > The inflows to the queue are less than the outflows, despite every worker spending 100% of its time processing tasks. Right, but at some point they will have to have SOME idle time. If they are NEVER idle the queue by definition is growing, not shrinking, as you said. Unless you have some system that monitors queue length and doesn't add jobs if it is like >n, which is what the article suggests (maximum queue size). But then you have to figure out what you do with those jobs that can't be added, which probably means just creating some kind of queue higher up in the pipe. If they are processing more than is being added, eventually things will run out of the queue.
- AceyMan 4y ago
  that's a great paper that clearly explains how "back pressure" [my term] at high utilization causes latency to go hockey-stick. (And thanks for the link bc I apparently forgot to bookmark it when I initially read it.)
  - roflyear 4y ago
    You're welcome! Important for people to get this I think bc we're using this tech more and more.