4 ms·
The context here is external queues like kafka. If you're writing stuff to a queue in your web app then it must be a relatively small load. This is different.
by kilgnad 4y ago
The context here is external queues like kafka. If you're writing stuff to a queue in your web app then it must be a relatively small load. This is different.
For analytics databases the batch ingestion requirement are usually so big that you have to save it to the file system hence the need for an external queue.
Heap allocated queues are different, at that stage I would say that it's in "processing" already and popped out of the external queue.
- ndriscoll 4y agoI guess "relatively small load" is... relative, but I've written that kind of thing to be able to handle ~70k sustained OLTP requests per second (including persistence to postgres) when load testing locally on my laptop. In any case the same thing applies to external queues. Your workers will often be more efficient if you pull off chunks of work to process together.
- kilgnad 4y agoBy small load I mean size. The frames in your batch jobs are tiny and few in number if you're getting 70k batch jobs per second. It's more similar to streaming... what you're doing here. In that case my velocity measurements are more applicable. You want your queues to be empty in general. A batch job is run every hour or something like that which is not what you're doing here. If you ran your load test for 5 minutes and you see your queues are 50 percent full. Well that means 10 minutes in you'll hit OOM. Assuming your load tests are at a constant rate. If your queues are mostly empty then it can handle the load you gave it and have room for spikes. It's just math.