I think it would be more accurate to say one can often break a problem into a deterministic one with commuting stages, producing a fork-join algorithm that woul

I think it would be more accurate to say one can often break a problem into a deterministic one with commuting stages, producing a fork-join algorithm that would be optimal if each step could be guaranteed to synchronously take exactly the same amount of time at each step, all hardware resources were truly saturated, and there was no communication overhead between parallel workers. For all of the above to be true, one would probably have to run the algorithm in serial on data that fit in L1, on a processor with no pipelining; in practice, a common strategy is to try to make the "horizontal" length of the stages larger (fitting problem instances into each stage, for example) to reduce the overall percentage of the time that's taken up by contention and communication overhead (or correlated problems like spurious aborts in the optimistic concurrency paradigm). Obviously, there are other tradeoffs that can be made to improve both latency and throughput (especially if you're dealing with a streaming problem and can employ latency hiding techniques like prefetching that exploit underutilized hardware), but I hope you'll agree that that particular sort of tradeoff (which is extremely common) does manifest as a latency-for-throughput-tradeoff; pipelining itself is a great example of that. Another way to say the same thing is that many algorithms improve throughput by deprioritizing fairness; usually extremely simple algorithms with small constants are going to beat more complicated ones with larger constants unless you make the problem sizes large enough, so to make those more complex algorithms useful you often have to "artificially" increase the problem size. > the one with better throughput might do better on tail latency as well, but under the same load, which is more important than latency under higher load, that it can handle. If algorithm A performs worse at "the same load" than algorithm B, what sane benchmark is going to say algorithm A has better latency? Either there's a point where that ceases to be the case, or we can just say B has better latency and throughput.