2 ms·
There's no need to wait for the flush interval to flush a full batch. Just do it. You'll get backpressure when your workers are all busy. Also, set your flush i
by ndriscoll 1y ago
There's no need to wait for the flush interval to flush a full batch. Just do it. You'll get backpressure when your workers are all busy. Also, set your flush interval closer to your throughput for your batch size. e.g. if you can do 100k/s, plan to flush a batch of 100 every ~2 ms. Then when you're not at peak load, you don't really pay for any extra latency for batching. As you approach peak load, you'll end up with full batches.
IIRC I got something like 70k rows/second on reddit dump data for an extended (couple hours) run using a similar approach with Scala on my i5-6600k (4 cores) with a JSON API that accepted 1 post at a time and batched internally. I think I used something like 2-4 workers, batch size 128, max flush of 5 ms. No COPY (the library I was using didn't have support for that, though I think it's not too bad to do manually with jdbc? I don't remember)