3 ms·
Queues can be full most of the time without the system being in a precarious state. Take for example an extremely spiky pattern of processing: once a day, a big
by Xylakant 4y ago
Queues can be full most of the time without the system being in a precarious state. Take for example an extremely spiky pattern of processing: once a day, a big batch of jobs is delivered. You know ahead of time how many items there are and have a pretty good handle on how long it takes to process one item. The items should be processed before the next batch arrives. You can then tune your system to empty the queue just before the next batch arrives. The queue is non-empty the majority of time, yet the system is perfectly stable.
Now, that is an extreme example, but similar behavior exists. It’s fine to have queues partially filled most of the time, as long as your average rate of processing exceeds the rate of incoming items by some margin.
- kilgnad 4y agoAS I said in the other post I am talking about the average and most likely use case. a flat and continuous rate of data processing and varied rates of data production. Spiking the processing is not what people think about with queues and is most likely NOT what the author of the article is talking about. Typically there's no reason why you would spike your processing unless such processing doesn't scale linearly with the amount of items processed (as the author in a sibling post to yours mentioned). Such a case must be deliberately introduced into the discussion as an exception to the model because it's not what people think about when discussing this. Or maybe it is given that 3 people brought the same exact exception up. Welp if that's the case then I can only say the author of this article and I certainly aren't referring to this use case. edit: I'm rate limited. So I'm referencing another post in my response that you won't see until the rate limit dies and I post it.