4 ms·
This is a great little article. I find too many programmers don't understand the value of batches, at every level of code. Batching is super important, both to
by asdfasgasdgasdg 7y ago
This is a great little article. I find too many programmers don't understand the value of batches, at every level of code. Batching is super important, both to amortize fixed costs and to reduce serialized blocking for "IO" (this case, if we define IO as "operations which fetch data from a slower data source than the one upon which we're operating").
There's another mistake programmers are prone to make after learning about the importance of batching. If batches of ten are good, why not a hundred? Why not a million? Lots of people make the mistake, after learning the value of batches, of assuming that bigger is always better. This isn't the case either. There are a few reasons why too large batches can also cause problems. For streaming work, large batches hurt latencies of initial results. Access locality can also be harmed when batches are too big. If your cache can fit a hundred of your objects and you perform operations batch-wise in batches of a thousand, you have to fetch each object from main memory per batch. If you worked in units of a hundred instead, then you would fetch each object from main memory once.
I find there are a couple of good rules of thumb for batching. Batches should be as small as possible, as long as they are large enough to solve the issues I mentioned earlier. I like to target batch sizes that are large enough that the fixed costs are no more than about 5% of the total costs of executing the batch. In the case of CPU-local work like the one described in this article, you would pick batch size s.t. operation 0 and associated fetches have finished on batch element 0 before you loop back to it to begin work on operation 1. In network situations where tracking batch completion and context switching is less expensive, this doesn't make as much sense. Just make it so that multiple batches can run concurrently, and issue multiple network requests to handle each one.
- magicalhippo 7y agoAt work I hit this issue just the other week. Customers reported performance issues. The issue was that they were pushing more data than we expected, but then again we should have expected that... Some intricate core code had to do like 10-15 queries per invoice line. Each is quick, but it adds up when you have 10k lines. The code my coworker wrote to do this was written in a per-line way. Now, since this code was called from other parts of our program, where it might just operate on a single line, I couldn't just cache willy-nilly inside the module. If so I'd have to introduce some calls to invalidate the caches or whatnot. Easy to miss and easy to screw up and get stale data. However, if instead the API had been batch oriented from the start, it would still work fine on a single line, but also allow for more optimizations for actual batches. The cases where it was called for a single line at a time was when the user was punching through, so being sub-optimal for that would not be an issue. Gonna be fun trying to disentangle this and try to make it batch friendly...
- triztian 7y agoSeems like a good option would be to implement a second version that has the batch API but leave the line oriented one as is for use by the callers expect it to behave in a per-line oriented way.
- magicalhippo 7y agoYeah, still won't be easy to integrate the batch optimizations now, since all the code in this module is written with the single-line-at-a-time assumption.
- spockz 7y agoIt seems to me that this would be an area where something like reactive streams excel at. The interface is by default streaming and the code dealing with the database could decide to buffer the incoming stream to an optimal amount for creating a query. (Of course only buffering until a maximum latency was reached.) This way neither the producer nor the consumer need to be aware of the exact buffer requirements and the whole can be optimised using back pressure.
- magicalhippo 7y agoWould be interesting to try, sadly not any library support in the language we use.
- Retric 7y agoThat opens things up for a host of long term maintenance issues. It might still be worthwhile, but it’s worth a serious discussion at a minimum.
- vhvjkyhkogvv 7y agoIf the single line one delegates to the batch one it should solve most of them.