4 ms·

Thanks for this, that has helped me see something I missed. If you’re dealing with something like SQS and are receiving variable sized batches of messages, the

by orf 4y ago

Thanks for this, that has helped me see something I missed. If you’re dealing with something like SQS and are receiving variable sized batches of messages, the consumers can be classed as “100% utilised” (always processing a batch of messages with no empty receives) whilst the queue would not be considered 100% utilised (the average delivered batch size is always less than the max).

roflyear 4y ago
What issue are you trying to solve? The expense of polling? Obviously there is a line where you do not want a bunch of expensive consumers hanging around - but this isn't the place IMO to micro-optimize your costs. You shouldn't have like 20x the consumers you need but you shouldn't worry about having EXACTLY the correct amount. In your example, are you implying that if more jobs come in more workers get spawned? Is that specific to the Amazon service?
- orf 4y ago
  If a process is busy processing a task, it is busy. A process could receive N tasks in a batch and process them at the same time. In this scenario it is possible for all workers to be utilised whilst the queue is not “100% utilised” - which is a failure mode the link you shared explained: > And by induction, we can show that a system that’s forever at 100% utilization will exceed any finite queue size you care to pick The inflows to the queue are less than the outflows, despite every worker spending 100% of its time processing tasks. It is specific on the utilisation type though, I’m mainly thinking about ML inference that has a fairly fixed initial cost per inference but scales quite nicely with >1 input.
  - roflyear 4y ago
    You can't have 100% utilization of any processing capacity (ignore queues) without reaching this type of failure. 100% CPU utilization is essentially just meaning you filled up the queue on your worker server. > The inflows to the queue are less than the outflows, despite every worker spending 100% of its time processing tasks. Right, but at some point they will have to have SOME idle time. If they are NEVER idle the queue by definition is growing, not shrinking, as you said. Unless you have some system that monitors queue length and doesn't add jobs if it is like >n, which is what the article suggests (maximum queue size). But then you have to figure out what you do with those jobs that can't be added, which probably means just creating some kind of queue higher up in the pipe. If they are processing more than is being added, eventually things will run out of the queue.

by orf 4y ago

roflyear 4y ago

What issue are you trying to solve? The expense of polling? Obviously there is a line where you do not want a bunch of expensive consumers hanging around - but this isn't the place IMO to micro-optimize your costs. You shouldn't have like 20x the consumers you need but you shouldn't worry about having EXACTLY the correct amount. In your example, are you implying that if more jobs come in more workers get spawned? Is that specific to the Amazon service?

orf 4y ago
If a process is busy processing a task, it is busy. A process could receive N tasks in a batch and process them at the same time. In this scenario it is possible for all workers to be utilised whilst the queue is not “100% utilised” - which is a failure mode the link you shared explained: > And by induction, we can show that a system that’s forever at 100% utilization will exceed any finite queue size you care to pick The inflows to the queue are less than the outflows, despite every worker spending 100% of its time processing tasks. It is specific on the utilisation type though, I’m mainly thinking about ML inference that has a fairly fixed initial cost per inference but scales quite nicely with >1 input.
- roflyear 4y ago
  You can't have 100% utilization of any processing capacity (ignore queues) without reaching this type of failure. 100% CPU utilization is essentially just meaning you filled up the queue on your worker server. > The inflows to the queue are less than the outflows, despite every worker spending 100% of its time processing tasks. Right, but at some point they will have to have SOME idle time. If they are NEVER idle the queue by definition is growing, not shrinking, as you said. Unless you have some system that monitors queue length and doesn't add jobs if it is like >n, which is what the article suggests (maximum queue size). But then you have to figure out what you do with those jobs that can't be added, which probably means just creating some kind of queue higher up in the pipe. If they are processing more than is being added, eventually things will run out of the queue.