3 ms·

No. That is not what I'm saying. I'm saying on average queues must be empty most of the time. And if you are seeing queues that are filled most of the time, you

No. That is not what I'm saying. I'm saying on average queues must be empty most of the time. And if you are seeing queues that are filled most of the time, your entire system is operating on the border of failure. This is true even when you have a big ass queue. The system must on average be more empty then it is filled. This is not to say that your system has to never have spikes in data. Your system can often have spikes, but this cannot ever be the norm as in the amount of spikes in data production rate cannot push the average production rate to exceed the average consumption rate.

orf 4y ago
I don’t see how this holds true. Let’s say I have a process that takes 1 hour to process a batch of items up to 1000 (I.e doesn’t scale linearly with input size). So, 1 hour to process 1 item and the same 1 hour to process 1000 items. If I have 1 message per minute being added to the queue, I can safely run 2 of these batch processes once a day to clear the queue. Or more generally can launch (queue_size / 1000) tasks once a day. Because surely it doesn’t matter about the queue being empty, what matters is the rate at which messages can be processed. With 2 processes running once a day the rate would be 2000/24/60 = 1.3 messages per minute in aggregate. The queue can be filled 23 hours per day without issue and without being on the brink of a failure state. Edit: you edited your comment whilst I was replying to specifically mention consumption rate. The aggregate consumption rate is all that matters, but it doesn’t follow that the queue “has to be more empty than not empty”. Those are different issues.
- kilgnad 4y ago
  You're talking about a different scenario. I'm talking about continuous inflow and outflow. Where outflow is processing (fixed rate) and inflow is traffic (variable). This is the classic assumption. In this state the space used in the queue is an indicator of mismatched consumption and production velocities. You're talking about a single batch job that runs once a day and empties the queue at the same rate no matter how much data is in the queue. Thus the more data you grab in batch the more efficient the processing so you grab data once per day instead of continuosly. This is a clear exception from the assumed norm that needs to be mentioned. That is not to say that your case is invalid. Many analytic databases possess your ingestion parameters. But the case is exceptional enough that if not explicitly mentioned it can be assumed that such a batch job is not a primitive in the model being discussed.
  - ndriscoll 4y ago
    It's not really that exceptional. You can do similar things with e.g. batch flushes every 5 ms for synchronous web requests and the same ideas will apply. This can give a massive increase in throughput while possibly decreasing overall response time for OLTP.
    - kilgnad 4y ago
      It's common, but exceptional enough that it should be mentioned because it is a specialized database that does batch ingestion. Let's say you flew from one city to another. People assume you took a plane. If you took a helicopter, though helicopters are common, it should be mentioned otherwise it won't be assumed.
      - ndriscoll 4y ago
        It doesn't take a special database though. You can do this sort of thing in your application in like 10-20 lines of code (assuming you have a framework with queues and promises). It works great with mysql/postgresql which both benefit dramatically from batching queries.
        kilgnad 4y ago
        The context here is external queues like kafka. If you're writing stuff to a queue in your web app then it must be a relatively small load. This is different. For analytics databases the batch ingestion requirement are usually so big that you have to save it to the file system hence the need for an external queue. Heap allocated queues are different, at that stage I would say that it's in "processing" already and popped out of the external queue.
        ndriscoll 4y ago
        I guess "relatively small load" is... relative, but I've written that kind of thing to be able to handle ~70k sustained OLTP requests per second (including persistence to postgres) when load testing locally on my laptop. In any case the same thing applies to external queues. Your workers will often be more efficient if you pull off chunks of work to process together.
Xylakant 4y ago
Queues can be full most of the time without the system being in a precarious state. Take for example an extremely spiky pattern of processing: once a day, a big batch of jobs is delivered. You know ahead of time how many items there are and have a pretty good handle on how long it takes to process one item. The items should be processed before the next batch arrives. You can then tune your system to empty the queue just before the next batch arrives. The queue is non-empty the majority of time, yet the system is perfectly stable. Now, that is an extreme example, but similar behavior exists. It’s fine to have queues partially filled most of the time, as long as your average rate of processing exceeds the rate of incoming items by some margin.
- kilgnad 4y ago
  AS I said in the other post I am talking about the average and most likely use case. a flat and continuous rate of data processing and varied rates of data production. Spiking the processing is not what people think about with queues and is most likely NOT what the author of the article is talking about. Typically there's no reason why you would spike your processing unless such processing doesn't scale linearly with the amount of items processed (as the author in a sibling post to yours mentioned). Such a case must be deliberately introduced into the discussion as an exception to the model because it's not what people think about when discussing this. Or maybe it is given that 3 people brought the same exact exception up. Welp if that's the case then I can only say the author of this article and I certainly aren't referring to this use case. edit: I'm rate limited. So I'm referencing another post in my response that you won't see until the rate limit dies and I post it.

orf 4y ago

I don’t see how this holds true. Let’s say I have a process that takes 1 hour to process a batch of items up to 1000 (I.e doesn’t scale linearly with input size). So, 1 hour to process 1 item and the same 1 hour to process 1000 items. If I have 1 message per minute being added to the queue, I can safely run 2 of these batch processes once a day to clear the queue. Or more generally can launch (queue_size / 1000) tasks once a day. Because surely it doesn’t matter about the queue being empty, what matters is the rate at which messages can be processed. With 2 processes running once a day the rate would be 2000/24/60 = 1.3 messages per minute in aggregate. The queue can be filled 23 hours per day without issue and without being on the brink of a failure state. Edit: you edited your comment whilst I was replying to specifically mention consumption rate. The aggregate consumption rate is all that matters, but it doesn’t follow that the queue “has to be more empty than not empty”. Those are different issues.

kilgnad 4y ago
You're talking about a different scenario. I'm talking about continuous inflow and outflow. Where outflow is processing (fixed rate) and inflow is traffic (variable). This is the classic assumption. In this state the space used in the queue is an indicator of mismatched consumption and production velocities. You're talking about a single batch job that runs once a day and empties the queue at the same rate no matter how much data is in the queue. Thus the more data you grab in batch the more efficient the processing so you grab data once per day instead of continuosly. This is a clear exception from the assumed norm that needs to be mentioned. That is not to say that your case is invalid. Many analytic databases possess your ingestion parameters. But the case is exceptional enough that if not explicitly mentioned it can be assumed that such a batch job is not a primitive in the model being discussed.
- ndriscoll 4y ago
  It's not really that exceptional. You can do similar things with e.g. batch flushes every 5 ms for synchronous web requests and the same ideas will apply. This can give a massive increase in throughput while possibly decreasing overall response time for OLTP.
  - kilgnad 4y ago
    It's common, but exceptional enough that it should be mentioned because it is a specialized database that does batch ingestion. Let's say you flew from one city to another. People assume you took a plane. If you took a helicopter, though helicopters are common, it should be mentioned otherwise it won't be assumed.
    - ndriscoll 4y ago
      It doesn't take a special database though. You can do this sort of thing in your application in like 10-20 lines of code (assuming you have a framework with queues and promises). It works great with mysql/postgresql which both benefit dramatically from batching queries.
      - kilgnad 4y ago
        The context here is external queues like kafka. If you're writing stuff to a queue in your web app then it must be a relatively small load. This is different. For analytics databases the batch ingestion requirement are usually so big that you have to save it to the file system hence the need for an external queue. Heap allocated queues are different, at that stage I would say that it's in "processing" already and popped out of the external queue.
        ndriscoll 4y ago
        I guess "relatively small load" is... relative, but I've written that kind of thing to be able to handle ~70k sustained OLTP requests per second (including persistence to postgres) when load testing locally on my laptop. In any case the same thing applies to external queues. Your workers will often be more efficient if you pull off chunks of work to process together.

Xylakant 4y ago

Queues can be full most of the time without the system being in a precarious state. Take for example an extremely spiky pattern of processing: once a day, a big batch of jobs is delivered. You know ahead of time how many items there are and have a pretty good handle on how long it takes to process one item. The items should be processed before the next batch arrives. You can then tune your system to empty the queue just before the next batch arrives. The queue is non-empty the majority of time, yet the system is perfectly stable. Now, that is an extreme example, but similar behavior exists. It’s fine to have queues partially filled most of the time, as long as your average rate of processing exceeds the rate of incoming items by some margin.

kilgnad 4y ago
AS I said in the other post I am talking about the average and most likely use case. a flat and continuous rate of data processing and varied rates of data production. Spiking the processing is not what people think about with queues and is most likely NOT what the author of the article is talking about. Typically there's no reason why you would spike your processing unless such processing doesn't scale linearly with the amount of items processed (as the author in a sibling post to yours mentioned). Such a case must be deliberately introduced into the discussion as an exception to the model because it's not what people think about when discussing this. Or maybe it is given that 3 people brought the same exact exception up. Welp if that's the case then I can only say the author of this article and I certainly aren't referring to this use case. edit: I'm rate limited. So I'm referencing another post in my response that you won't see until the rate limit dies and I post it.