4 ms·

It's common, but exceptional enough that it should be mentioned because it is a specialized database that does batch ingestion. Let's say you flew from one cit

It's common, but exceptional enough that it should be mentioned because it is a specialized database that does batch ingestion. Let's say you flew from one city to another. People assume you took a plane. If you took a helicopter, though helicopters are common, it should be mentioned otherwise it won't be assumed.

ndriscoll 4y ago
It doesn't take a special database though. You can do this sort of thing in your application in like 10-20 lines of code (assuming you have a framework with queues and promises). It works great with mysql/postgresql which both benefit dramatically from batching queries.
- kilgnad 4y ago
  The context here is external queues like kafka. If you're writing stuff to a queue in your web app then it must be a relatively small load. This is different. For analytics databases the batch ingestion requirement are usually so big that you have to save it to the file system hence the need for an external queue. Heap allocated queues are different, at that stage I would say that it's in "processing" already and popped out of the external queue.
  - ndriscoll 4y ago
    I guess "relatively small load" is... relative, but I've written that kind of thing to be able to handle ~70k sustained OLTP requests per second (including persistence to postgres) when load testing locally on my laptop. In any case the same thing applies to external queues. Your workers will often be more efficient if you pull off chunks of work to process together.
    - kilgnad 4y ago
      By small load I mean size. The frames in your batch jobs are tiny and few in number if you're getting 70k batch jobs per second. It's more similar to streaming... what you're doing here. In that case my velocity measurements are more applicable. You want your queues to be empty in general. A batch job is run every hour or something like that which is not what you're doing here. If you ran your load test for 5 minutes and you see your queues are 50 percent full. Well that means 10 minutes in you'll hit OOM. Assuming your load tests are at a constant rate. If your queues are mostly empty then it can handle the load you gave it and have room for spikes. It's just math.

ndriscoll 4y ago

It doesn't take a special database though. You can do this sort of thing in your application in like 10-20 lines of code (assuming you have a framework with queues and promises). It works great with mysql/postgresql which both benefit dramatically from batching queries.

kilgnad 4y ago
The context here is external queues like kafka. If you're writing stuff to a queue in your web app then it must be a relatively small load. This is different. For analytics databases the batch ingestion requirement are usually so big that you have to save it to the file system hence the need for an external queue. Heap allocated queues are different, at that stage I would say that it's in "processing" already and popped out of the external queue.
- ndriscoll 4y ago
  I guess "relatively small load" is... relative, but I've written that kind of thing to be able to handle ~70k sustained OLTP requests per second (including persistence to postgres) when load testing locally on my laptop. In any case the same thing applies to external queues. Your workers will often be more efficient if you pull off chunks of work to process together.
  - kilgnad 4y ago
    By small load I mean size. The frames in your batch jobs are tiny and few in number if you're getting 70k batch jobs per second. It's more similar to streaming... what you're doing here. In that case my velocity measurements are more applicable. You want your queues to be empty in general. A batch job is run every hour or something like that which is not what you're doing here. If you ran your load test for 5 minutes and you see your queues are 50 percent full. Well that means 10 minutes in you'll hit OOM. Assuming your load tests are at a constant rate. If your queues are mostly empty then it can handle the load you gave it and have room for spikes. It's just math.