2 ms·
Batch processing is always easier to tune performance over real time one. Classic example: Batch upserting legacy data into new service. Instead of looping an
by revskill 4y ago
Batch processing is always easier to tune performance over real time one.
Classic example: Batch upserting legacy data into new service.
Instead of looping and try to search for existing items before insertion, we could search first, then insert in batch!
- necovek 4y agoIt's not always easier: the requirements are frequently different. Live processing requires progress updates for the user, and if it takes too long, is often relegated to batch jobs: other than that, it's equally easy or hard to optimize them for identical (absolute) time thresholds. Batch jobs are similarly prone to races, starvation and other issues: you can't simply do a huge "upsert" without locking the full table, because it might lock other parts of the system, including other batch jobs accessing the same data.
- revskill 4y agoThe point is "easier to tune performance", not only "easier". In real time processing, you have limited choice of course ?
- necovek 4y agoI don't see it: it's all exactly the same. If your performance goal is not achievable in time required for live processing (whatever you decide that to be), you just relegate to a background job and then it's the same as any other batch job. Sure, this is a slight complexity on top, but it's such a common and well known pattern that it doesn't really add any practical complexity after you do it once. It's a fallacy to think that it's any different.