The author's thesis largely holds for systems that are at least reasonably well implemented. It's easy to do things that hurt both latency and throughput. You c

The author's thesis largely holds for systems that are at least reasonably well implemented. It's easy to do things that hurt both latency and throughput. You can just waste lots of CPU or memory. You can add unnecessary coordination bottlenecks. You can use inefficient algorithms. Therefore, it follows that you can improve both latency and throughput by fixing these implementation details. Quite often, this will be sufficient for both latency- and throughput-sensitive users. Of particular note: even if you're already operating in a distributed, partitioned etc. environment, improving per-node efficiency often means that you can serve a given workload with fewer nodes and reduce scaling costs. In the extreme case, the less efficient system would require so many nodes even if it scaled linearly that the more-than-linear coordination costs would kill it entirely. In other words, there's always an absolute limit on how "big" your system can grow. To raise that limit you must often address both per-node performance and scaling efficiency. This is not merely hypothetical. When I was working on distributed filesystems, a competing system that was genuinely more scalable in terms of node count was so ruinously inefficient on each node that they not only couldn't beat us on latency but they basically couldn't beat us on throughput per dollar at any level either. So they resorted to politics instead of engineering, but that's another topic. The key point here is that eliminating waste will typically improve both metrics while adding scalability will only improve one at most and might actually not even raise the ceiling on that. Once a system is "well implemented" as I've described, the OP's tradeoff becomes relevant. But TBH a lot of systems just never get to that point. They end up being good enough for business purposes even when there's still tons of room for technical improvement. Often, some simple math around utilization levels and controls around batch sizes will suffice.