4 ms·

Wrong. Your thinking is where the nuance is lacking. As I said you need to think of the system in terms of it's derivative. The rate of data consumed must on av

Wrong. Your thinking is where the nuance is lacking. As I said you need to think of the system in terms of it's derivative. The rate of data consumed must on average be faster then the production. The net difference of these rates must equal a negative velocity with the rate of data going in as a positive number and data consumed as negative. There is no system on the face of the earth that can operate at an average positive velocity. That is categorical failure. This is mathematically and logically true no matter how big your queue is. Even for the situation your describing... A really big ass queue, the average velocity of that system must still be negative. Unless your queue is so big that it can withstand an average negative velocity over the lifetime of the system which is kind of absurd. Most of the time your queues should be empty. If your queues are observationally often filled with data but not full it means you're still operating at a negative velocity but you've tuned your system to operate at the border of failure. It means your system hits unsustainable velocities often. You may want this for cost reasons but but such a system is not robust enough for my comfort. The queue should be empty most of the time with occasional spikes at most.

zokier 4y ago
> You may want this for cost reasons but but such a system is not robust enough for my comfort Your comfort is secondary to the business needs that the system is designed to fulfill. In practice some risks are better to accept than mitigate
- kilgnad 4y ago
  My comfort is a measure of business aversion to risk. It depends on the business of our respective industries. Either way you have admitted that you tuned the system to operate with an acceptance of risk. Likely for cost reasons. This is fine, if you deliberately chose to do this.
  - zokier 4y ago
    > This is fine, if you deliberately chose to do this. Yet you bravely asserted that > At best you can account for occasional spikes as the article stated but a queue even half full should never be the operational norm. Suddenly this thing that you said "should never be" changed to "is fine"
    - kilgnad 4y ago
      When humans communicate. We don't talk about exceptions where you deliberately choose to operate outside of the norm. If you deliberately choose to drive a car without a seat belt knowing this risk. This is fine. You want to sky dive while deliberately choosing not to use a parachute? That's fine. But don't expect me to adjust my communication to account for your exception. You chose to do this. But let's not pretend that you don't know about this fact in human communication. It's an obvious thing you know. Yet you chose to characterize your post in a way that negatively makes my statement look unreasonable and your actions look normal. Please don't do this. If you choose to sky dive without a parachute, that's fine, but admit you are operating with risk for the sake of cost. Do not try mischaracterize my intentions, that is just rude.
orf 4y ago
This is a very long-winded way of saying “you should have enough capacity to process all items in the queue before the queue fills up and everything explodes”. I agree. But that’s not the argument the post is making, nor does it support the position that “queues should be empty”. No, they shouldn’t. It’s fine to let a queue build up (and thus not be empty), then process all the items in the queue in a short time frame. There are efficiency savings if you are able to do this.
- kilgnad 4y ago
  No. That is not what I'm saying. I'm saying on average queues must be empty most of the time. And if you are seeing queues that are filled most of the time, your entire system is operating on the border of failure. This is true even when you have a big ass queue. The system must on average be more empty then it is filled. This is not to say that your system has to never have spikes in data. Your system can often have spikes, but this cannot ever be the norm as in the amount of spikes in data production rate cannot push the average production rate to exceed the average consumption rate.
  - orf 4y ago
    I don’t see how this holds true. Let’s say I have a process that takes 1 hour to process a batch of items up to 1000 (I.e doesn’t scale linearly with input size). So, 1 hour to process 1 item and the same 1 hour to process 1000 items. If I have 1 message per minute being added to the queue, I can safely run 2 of these batch processes once a day to clear the queue. Or more generally can launch (queue_size / 1000) tasks once a day. Because surely it doesn’t matter about the queue being empty, what matters is the rate at which messages can be processed. With 2 processes running once a day the rate would be 2000/24/60 = 1.3 messages per minute in aggregate. The queue can be filled 23 hours per day without issue and without being on the brink of a failure state. Edit: you edited your comment whilst I was replying to specifically mention consumption rate. The aggregate consumption rate is all that matters, but it doesn’t follow that the queue “has to be more empty than not empty”. Those are different issues.
    - 4y ago

zokier 4y ago

> You may want this for cost reasons but but such a system is not robust enough for my comfort Your comfort is secondary to the business needs that the system is designed to fulfill. In practice some risks are better to accept than mitigate

kilgnad 4y ago
My comfort is a measure of business aversion to risk. It depends on the business of our respective industries. Either way you have admitted that you tuned the system to operate with an acceptance of risk. Likely for cost reasons. This is fine, if you deliberately chose to do this.
- zokier 4y ago
  > This is fine, if you deliberately chose to do this. Yet you bravely asserted that > At best you can account for occasional spikes as the article stated but a queue even half full should never be the operational norm. Suddenly this thing that you said "should never be" changed to "is fine"
  - kilgnad 4y ago
    When humans communicate. We don't talk about exceptions where you deliberately choose to operate outside of the norm. If you deliberately choose to drive a car without a seat belt knowing this risk. This is fine. You want to sky dive while deliberately choosing not to use a parachute? That's fine. But don't expect me to adjust my communication to account for your exception. You chose to do this. But let's not pretend that you don't know about this fact in human communication. It's an obvious thing you know. Yet you chose to characterize your post in a way that negatively makes my statement look unreasonable and your actions look normal. Please don't do this. If you choose to sky dive without a parachute, that's fine, but admit you are operating with risk for the sake of cost. Do not try mischaracterize my intentions, that is just rude.

orf 4y ago

This is a very long-winded way of saying “you should have enough capacity to process all items in the queue before the queue fills up and everything explodes”. I agree. But that’s not the argument the post is making, nor does it support the position that “queues should be empty”. No, they shouldn’t. It’s fine to let a queue build up (and thus not be empty), then process all the items in the queue in a short time frame. There are efficiency savings if you are able to do this.

kilgnad 4y ago
No. That is not what I'm saying. I'm saying on average queues must be empty most of the time. And if you are seeing queues that are filled most of the time, your entire system is operating on the border of failure. This is true even when you have a big ass queue. The system must on average be more empty then it is filled. This is not to say that your system has to never have spikes in data. Your system can often have spikes, but this cannot ever be the norm as in the amount of spikes in data production rate cannot push the average production rate to exceed the average consumption rate.
- orf 4y ago
  I don’t see how this holds true. Let’s say I have a process that takes 1 hour to process a batch of items up to 1000 (I.e doesn’t scale linearly with input size). So, 1 hour to process 1 item and the same 1 hour to process 1000 items. If I have 1 message per minute being added to the queue, I can safely run 2 of these batch processes once a day to clear the queue. Or more generally can launch (queue_size / 1000) tasks once a day. Because surely it doesn’t matter about the queue being empty, what matters is the rate at which messages can be processed. With 2 processes running once a day the rate would be 2000/24/60 = 1.3 messages per minute in aggregate. The queue can be filled 23 hours per day without issue and without being on the brink of a failure state. Edit: you edited your comment whilst I was replying to specifically mention consumption rate. The aggregate consumption rate is all that matters, but it doesn’t follow that the queue “has to be more empty than not empty”. Those are different issues.
  - 4y ago