5 ms·

Wrong. They should. Any queue that fills up, say halfway, is an indication the system is in a precarious state. Its simple. Queues have limited space if the v

Wrong. They should. Any queue that fills up, say halfway, is an indication the system is in a precarious state. Its simple. Queues have limited space if the velocity of data going in is faster then data going out the queue will fill up until it's full. A half full queue indicates the velocity is in such a state and such a state is unsustainable. You need to build your system such that it stays out of unsustainable states as much as possible. If you're finding that your queue fills up half way a lot that means you're building your system in such a way that it operates at the border of sustainability. In other words the system often ingests data at rates that cannot be sustained indefinitely. If a system with a queue half full operates in that same state twice as long, you now have a full queue and you're losing data. In essence your queues should mostly be empty most of the time. At best you can account for occasional spikes as the article stated but a queue even half full should never be the operational norm.

orf 4y ago
This response lacks nuance and focuses too much on the operational side of queues. Not all queues are “realistically bounded”. Let’s take a hypothetical silly “order queue” where each entry represents a €10 item that’s been purchased. The queue feeds into a boxing machine that puts the item in a box and posts it to the user. The point at which a queue hits an upper bound would represent billions of euros of orders. That’s not really a realistic failure mode to hit without noticing long beforehand. Meanwhile, if you always want your queue empty then your boxing machine is going to be shipping thousands of small boxes because the moment an order arrives, it’s processed into a box. It’s much more efficient to have a small, managed and time-bounded backlog of orders that the boxing machine uses to batch orders together in a smaller number of larger boxes.
- kilgnad 4y ago
  Wrong. Your thinking is where the nuance is lacking. As I said you need to think of the system in terms of it's derivative. The rate of data consumed must on average be faster then the production. The net difference of these rates must equal a negative velocity with the rate of data going in as a positive number and data consumed as negative. There is no system on the face of the earth that can operate at an average positive velocity. That is categorical failure. This is mathematically and logically true no matter how big your queue is. Even for the situation your describing... A really big ass queue, the average velocity of that system must still be negative. Unless your queue is so big that it can withstand an average negative velocity over the lifetime of the system which is kind of absurd. Most of the time your queues should be empty. If your queues are observationally often filled with data but not full it means you're still operating at a negative velocity but you've tuned your system to operate at the border of failure. It means your system hits unsustainable velocities often. You may want this for cost reasons but but such a system is not robust enough for my comfort. The queue should be empty most of the time with occasional spikes at most.
  - zokier 4y ago
    > You may want this for cost reasons but but such a system is not robust enough for my comfort Your comfort is secondary to the business needs that the system is designed to fulfill. In practice some risks are better to accept than mitigate
    - kilgnad 4y ago
      My comfort is a measure of business aversion to risk. It depends on the business of our respective industries. Either way you have admitted that you tuned the system to operate with an acceptance of risk. Likely for cost reasons. This is fine, if you deliberately chose to do this.
      - zokier 4y ago
        > This is fine, if you deliberately chose to do this. Yet you bravely asserted that > At best you can account for occasional spikes as the article stated but a queue even half full should never be the operational norm. Suddenly this thing that you said "should never be" changed to "is fine"
        kilgnad 4y ago
        When humans communicate. We don't talk about exceptions where you deliberately choose to operate outside of the norm. If you deliberately choose to drive a car without a seat belt knowing this risk. This is fine. You want to sky dive while deliberately choosing not to use a parachute? That's fine. But don't expect me to adjust my communication to account for your exception. You chose to do this. But let's not pretend that you don't know about this fact in human communication. It's an obvious thing you know. Yet you chose to characterize your post in a way that negatively makes my statement look unreasonable and your actions look normal. Please don't do this. If you choose to sky dive without a parachute, that's fine, but admit you are operating with risk for the sake of cost. Do not try mischaracterize my intentions, that is just rude.
  - orf 4y ago
zokier 4y ago
> You need to build your system such that it stays out of unsustainable states as much as possible. You assert that without justification. Typical counter-example would be system where you have predictable daily load-pattern, it can be perfectly reasonable to build the system that accumulates items during peak hours and clears queues during quiet hours.
dooglius 4y ago
You are making a somewhat subtle error here, which is assuming that the (output_rate-input_rate) is statistically independent of the size of the queue. Note that this does not necessarily imply that the output process changes based on the size of the queue. For example, consider a microphone that emits 100 raw samples to a buffer per ms and a real-time thread scheduled every 2ms that processes very quickly. a 300-sample buffer is more than enough here, even though the buffer will be over half full about a quarter of the time, and will never be empty. In this case, the velocity conditional on size being less than half is -100 samples/ms and the velocity conditional on size being greater than half is 400 samples/ms.
- kilgnad 4y ago
  This example suffers from the same problem as the other counter examples. Why are you deliberately slowing down the processing to only happen at 2ms intervals? If you have continuous processing the queue will be empty always assuming processing is faster then data production. You're maybe referring to what the other person mentioned... processing that does not scale with the amount of data processed so you want the processing to operate on the biggest batch of data possible. This is, while a common use case, not common enough that it should be assumed without be mentioned. However, in your example it seems to be happening on the same computer (you mentioned mic), (and thus not web, which is what's the assumed context here) therefore data production rate doesn't seem to spike. If your processing is faster then data production you don't even need a queue, just stream each frame straight to processing in a single thread. But this also assumes the nature of your processing. The details for the nature of the processing can change things to require a queue for something like the averages of frames at discreet intervals... but that's different from the use case of queues like kafka or zmq for which I think is the main context here. Edit: I reference other responses here, but you won't see them immediately. I'm currently rate limited and I have those replies in a queue so those will be posted later today.

orf 4y ago

This response lacks nuance and focuses too much on the operational side of queues. Not all queues are “realistically bounded”. Let’s take a hypothetical silly “order queue” where each entry represents a €10 item that’s been purchased. The queue feeds into a boxing machine that puts the item in a box and posts it to the user. The point at which a queue hits an upper bound would represent billions of euros of orders. That’s not really a realistic failure mode to hit without noticing long beforehand. Meanwhile, if you always want your queue empty then your boxing machine is going to be shipping thousands of small boxes because the moment an order arrives, it’s processed into a box. It’s much more efficient to have a small, managed and time-bounded backlog of orders that the boxing machine uses to batch orders together in a smaller number of larger boxes.

kilgnad 4y ago
Wrong. Your thinking is where the nuance is lacking. As I said you need to think of the system in terms of it's derivative. The rate of data consumed must on average be faster then the production. The net difference of these rates must equal a negative velocity with the rate of data going in as a positive number and data consumed as negative. There is no system on the face of the earth that can operate at an average positive velocity. That is categorical failure. This is mathematically and logically true no matter how big your queue is. Even for the situation your describing... A really big ass queue, the average velocity of that system must still be negative. Unless your queue is so big that it can withstand an average negative velocity over the lifetime of the system which is kind of absurd. Most of the time your queues should be empty. If your queues are observationally often filled with data but not full it means you're still operating at a negative velocity but you've tuned your system to operate at the border of failure. It means your system hits unsustainable velocities often. You may want this for cost reasons but but such a system is not robust enough for my comfort. The queue should be empty most of the time with occasional spikes at most.
- zokier 4y ago
  > You may want this for cost reasons but but such a system is not robust enough for my comfort Your comfort is secondary to the business needs that the system is designed to fulfill. In practice some risks are better to accept than mitigate
  - kilgnad 4y ago
    My comfort is a measure of business aversion to risk. It depends on the business of our respective industries. Either way you have admitted that you tuned the system to operate with an acceptance of risk. Likely for cost reasons. This is fine, if you deliberately chose to do this.
    - zokier 4y ago
      > This is fine, if you deliberately chose to do this. Yet you bravely asserted that > At best you can account for occasional spikes as the article stated but a queue even half full should never be the operational norm. Suddenly this thing that you said "should never be" changed to "is fine"
      - kilgnad 4y ago
        When humans communicate. We don't talk about exceptions where you deliberately choose to operate outside of the norm. If you deliberately choose to drive a car without a seat belt knowing this risk. This is fine. You want to sky dive while deliberately choosing not to use a parachute? That's fine. But don't expect me to adjust my communication to account for your exception. You chose to do this. But let's not pretend that you don't know about this fact in human communication. It's an obvious thing you know. Yet you chose to characterize your post in a way that negatively makes my statement look unreasonable and your actions look normal. Please don't do this. If you choose to sky dive without a parachute, that's fine, but admit you are operating with risk for the sake of cost. Do not try mischaracterize my intentions, that is just rude.
- orf 4y ago

zokier 4y ago

> You need to build your system such that it stays out of unsustainable states as much as possible. You assert that without justification. Typical counter-example would be system where you have predictable daily load-pattern, it can be perfectly reasonable to build the system that accumulates items during peak hours and clears queues during quiet hours.

dooglius 4y ago

You are making a somewhat subtle error here, which is assuming that the (output_rate-input_rate) is statistically independent of the size of the queue. Note that this does not necessarily imply that the output process changes based on the size of the queue. For example, consider a microphone that emits 100 raw samples to a buffer per ms and a real-time thread scheduled every 2ms that processes very quickly. a 300-sample buffer is more than enough here, even though the buffer will be over half full about a quarter of the time, and will never be empty. In this case, the velocity conditional on size being less than half is -100 samples/ms and the velocity conditional on size being greater than half is 400 samples/ms.

kilgnad 4y ago
This example suffers from the same problem as the other counter examples. Why are you deliberately slowing down the processing to only happen at 2ms intervals? If you have continuous processing the queue will be empty always assuming processing is faster then data production. You're maybe referring to what the other person mentioned... processing that does not scale with the amount of data processed so you want the processing to operate on the biggest batch of data possible. This is, while a common use case, not common enough that it should be assumed without be mentioned. However, in your example it seems to be happening on the same computer (you mentioned mic), (and thus not web, which is what's the assumed context here) therefore data production rate doesn't seem to spike. If your processing is faster then data production you don't even need a queue, just stream each frame straight to processing in a single thread. But this also assumes the nature of your processing. The details for the nature of the processing can change things to require a queue for something like the averages of frames at discreet intervals... but that's different from the use case of queues like kafka or zmq for which I think is the main context here. Edit: I reference other responses here, but you won't see them immediately. I'm currently rate limited and I have those replies in a queue so those will be posted later today.