4 ms·

> My point is instead of doing 100k transactions in your web app, you should look at how to gather them into batches. This sounds odd to me. Assuming 100k inde

> My point is instead of doing 100k transactions in your web app, you should look at how to gather them into batches. This sounds odd to me. Assuming 100k independent web requests, a reasonable web app ought to be 100k transactions, and the database is the bottleneck. Suggesting that the web app should rework all this into one bulk transaction is ignoring the supposed value-add of the database abstraction and re-implementing it in the web app layer. And most attempts are probably going to do it poorly. They're going to fail at maintaining the coherence and reliability of the naive solution that leaves this work to the database. Of course, one ought to design appropriate bulk operations in a web app where there is really one client trying to accomplish batches of things. Then, you can also push the batch semantics up the stack, potentially all the way to the UX. But that's a lot different than going through heroics to merge independent client request streams into batches while telling yourself the database is not actually the bottleneck...

ndriscoll 3y ago
There aren't really heroics involved in doing this. e.g. in a promise-based framework, put your request onto a Queue[(RequestData, Promise)]. Await the promise as you would with a normal db request in your web handler. Have a db worker thread that pulls off the queue with a max wait time. Now pretend you received a batch file for your business logic. After processing the batch, complete the promises with their results. The basic scaffolding is maybe 20 LoC in a decent language/framework. Much easier than alternatives like sharding or microservices or eventual consistency anyway. Basic toy example I made a while back to demonstrate the technique using a Repository Pattern: https://gist.github.com/ndriscoll/6a1df10c8ea1a25c48c615c409da774f https://gist.github.com/ndriscoll/6a1df10c8ea1a25c48c615c409... I think somewhere I have a more built-out example with error handling and and where you have more than just "body" in the model. In the real world the business logic for processing the batch and error handling is going to be a decent amount more complicated. etc. etc. But the point is more about how the batching mechanic works.

ndriscoll 3y ago

There aren't really heroics involved in doing this. e.g. in a promise-based framework, put your request onto a Queue[(RequestData, Promise)]. Await the promise as you would with a normal db request in your web handler. Have a db worker thread that pulls off the queue with a max wait time. Now pretend you received a batch file for your business logic. After processing the batch, complete the promises with their results. The basic scaffolding is maybe 20 LoC in a decent language/framework. Much easier than alternatives like sharding or microservices or eventual consistency anyway. Basic toy example I made a while back to demonstrate the technique using a Repository Pattern: https://gist.github.com/ndriscoll/6a1df10c8ea1a25c48c615c409da774f https://gist.github.com/ndriscoll/6a1df10c8ea1a25c48c615c409... I think somewhere I have a more built-out example with error handling and and where you have more than just "body" in the model. In the real world the business logic for processing the batch and error handling is going to be a decent amount more complicated. etc. etc. But the point is more about how the batching mechanic works.