5 ms·
Right, but how do you do this concurrently? This sounds like you plan to sequentially execute these batches, which could be painfully slow.
by whakim 3y ago
Right, but how do you do this concurrently? This sounds like you plan to sequentially execute these batches, which could be painfully slow.
- jaggederest 3y agoYou can either limit the load on the database by doing it "painfully slow", or run it all at once, there's no third option. The batches can be arbitrarily large per-query - 5000; 50,000; 500,000 rows per sequential query; depending on how much your database can handle at once without being unacceptably overloaded. There's zero need for concurrency in this context - let the database handle the concurrency internally to the storage engine, and just specify the logical batch size. N.B. postgresql can use parallel execution of workers in batches above a certain size, Depending On Your Settings™ and version
- whakim 3y agoI have found that in most real-world cases if you parallelize SQL commands you can significantly increase CPU utilization and thus database throughput.