4 ms·

You're talking about a different scenario. I'm talking about continuous inflow and outflow. Where outflow is processing (fixed rate) and inflow is traffic (vari

You're talking about a different scenario. I'm talking about continuous inflow and outflow. Where outflow is processing (fixed rate) and inflow is traffic (variable). This is the classic assumption. In this state the space used in the queue is an indicator of mismatched consumption and production velocities. You're talking about a single batch job that runs once a day and empties the queue at the same rate no matter how much data is in the queue. Thus the more data you grab in batch the more efficient the processing so you grab data once per day instead of continuosly. This is a clear exception from the assumed norm that needs to be mentioned. That is not to say that your case is invalid. Many analytic databases possess your ingestion parameters. But the case is exceptional enough that if not explicitly mentioned it can be assumed that such a batch job is not a primitive in the model being discussed.

ndriscoll 4y ago
It's not really that exceptional. You can do similar things with e.g. batch flushes every 5 ms for synchronous web requests and the same ideas will apply. This can give a massive increase in throughput while possibly decreasing overall response time for OLTP.
- kilgnad 4y ago
  It's common, but exceptional enough that it should be mentioned because it is a specialized database that does batch ingestion. Let's say you flew from one city to another. People assume you took a plane. If you took a helicopter, though helicopters are common, it should be mentioned otherwise it won't be assumed.
  - ndriscoll 4y ago
    It doesn't take a special database though. You can do this sort of thing in your application in like 10-20 lines of code (assuming you have a framework with queues and promises). It works great with mysql/postgresql which both benefit dramatically from batching queries.
    - kilgnad 4y ago
      The context here is external queues like kafka. If you're writing stuff to a queue in your web app then it must be a relatively small load. This is different. For analytics databases the batch ingestion requirement are usually so big that you have to save it to the file system hence the need for an external queue. Heap allocated queues are different, at that stage I would say that it's in "processing" already and popped out of the external queue.
      - ndriscoll 4y ago
        I guess "relatively small load" is... relative, but I've written that kind of thing to be able to handle ~70k sustained OLTP requests per second (including persistence to postgres) when load testing locally on my laptop. In any case the same thing applies to external queues. Your workers will often be more efficient if you pull off chunks of work to process together.
        kilgnad 4y ago
        By small load I mean size. The frames in your batch jobs are tiny and few in number if you're getting 70k batch jobs per second. It's more similar to streaming... what you're doing here. In that case my velocity measurements are more applicable. You want your queues to be empty in general. A batch job is run every hour or something like that which is not what you're doing here. If you ran your load test for 5 minutes and you see your queues are 50 percent full. Well that means 10 minutes in you'll hit OOM. Assuming your load tests are at a constant rate. If your queues are mostly empty then it can handle the load you gave it and have room for spikes. It's just math.
orf 4y ago
I disagree it’s exceptional even if my examples have been caricatures. Very few technical problems don’t have any elasticity in outflows, but a lot of problems do benefit from a form of batching. Heck, even something as simple as inserting records into a non-analytic database benefits greatly from batching (inserting N rows rather than 1) and thus reducing TPS, or calling an external service, or writing files to some storage, or running inferences, or writing to a ledger, or anything. Batching, in all forms, exists absolutely everywhere from super-scalar CPU uops to storage to networking to the physical world. The exception is to do something without batching, even if the whole stack you’re using from the ground up is built on it. Given the common use case of batching, which inherently requires items to be queued for a period of time, how can you say that queues should always be empty? They would statistically rarely be empty. Which is a good thing. Because it enables batching. If you want to queue and insert 1000 records into a database with 1000 transactions across 1000 workers, then say your workload is highly optimised because there are never any records in your queue, cool. Not very true though.
- kilgnad 4y ago
  >Very few technical problems don’t have any elasticity in outflows, They have elasticity but you don't care for it. You run it at the maximum. As you mentioned the only case where you wouldn't is if you have some transaction that doesn't scale linearly with with frames/events. >Given the common use case of batching, which inherently requires items to be queued for a period of time, how can you say that queues should always be empty? They would statistically rarely be empty. Which is a good thing. Because it enables batching. In my experience, while batching is not uncommon, streaming is the much more common use case. So we can agree to disagree here.

ndriscoll 4y ago

It's not really that exceptional. You can do similar things with e.g. batch flushes every 5 ms for synchronous web requests and the same ideas will apply. This can give a massive increase in throughput while possibly decreasing overall response time for OLTP.

kilgnad 4y ago
It's common, but exceptional enough that it should be mentioned because it is a specialized database that does batch ingestion. Let's say you flew from one city to another. People assume you took a plane. If you took a helicopter, though helicopters are common, it should be mentioned otherwise it won't be assumed.
- ndriscoll 4y ago
  It doesn't take a special database though. You can do this sort of thing in your application in like 10-20 lines of code (assuming you have a framework with queues and promises). It works great with mysql/postgresql which both benefit dramatically from batching queries.
  - kilgnad 4y ago
    The context here is external queues like kafka. If you're writing stuff to a queue in your web app then it must be a relatively small load. This is different. For analytics databases the batch ingestion requirement are usually so big that you have to save it to the file system hence the need for an external queue. Heap allocated queues are different, at that stage I would say that it's in "processing" already and popped out of the external queue.
    - ndriscoll 4y ago
      I guess "relatively small load" is... relative, but I've written that kind of thing to be able to handle ~70k sustained OLTP requests per second (including persistence to postgres) when load testing locally on my laptop. In any case the same thing applies to external queues. Your workers will often be more efficient if you pull off chunks of work to process together.
      - kilgnad 4y ago
        By small load I mean size. The frames in your batch jobs are tiny and few in number if you're getting 70k batch jobs per second. It's more similar to streaming... what you're doing here. In that case my velocity measurements are more applicable. You want your queues to be empty in general. A batch job is run every hour or something like that which is not what you're doing here. If you ran your load test for 5 minutes and you see your queues are 50 percent full. Well that means 10 minutes in you'll hit OOM. Assuming your load tests are at a constant rate. If your queues are mostly empty then it can handle the load you gave it and have room for spikes. It's just math.

orf 4y ago

I disagree it’s exceptional even if my examples have been caricatures. Very few technical problems don’t have any elasticity in outflows, but a lot of problems do benefit from a form of batching. Heck, even something as simple as inserting records into a non-analytic database benefits greatly from batching (inserting N rows rather than 1) and thus reducing TPS, or calling an external service, or writing files to some storage, or running inferences, or writing to a ledger, or anything. Batching, in all forms, exists absolutely everywhere from super-scalar CPU uops to storage to networking to the physical world. The exception is to do something without batching, even if the whole stack you’re using from the ground up is built on it. Given the common use case of batching, which inherently requires items to be queued for a period of time, how can you say that queues should always be empty? They would statistically rarely be empty. Which is a good thing. Because it enables batching. If you want to queue and insert 1000 records into a database with 1000 transactions across 1000 workers, then say your workload is highly optimised because there are never any records in your queue, cool. Not very true though.

kilgnad 4y ago
>Very few technical problems don’t have any elasticity in outflows, They have elasticity but you don't care for it. You run it at the maximum. As you mentioned the only case where you wouldn't is if you have some transaction that doesn't scale linearly with with frames/events. >Given the common use case of batching, which inherently requires items to be queued for a period of time, how can you say that queues should always be empty? They would statistically rarely be empty. Which is a good thing. Because it enables batching. In my experience, while batching is not uncommon, streaming is the much more common use case. So we can agree to disagree here.