3 ms·

The Latency/Throughput Tradeoff: Why Fast Services Are Slow and Vice Versa

25 points by vinnyglennon 4y ago

namelosw 4y ago
1. It's a trade-off so real-world implementations are likely to suck at both ends of the spectrum. (eg. Old Ruby servers are neither responsive nor they have large throughput) 2. The trade-off is more about computation in general. In typical "service" scenarios you almost always want to prioritize latency over throughput, while in most "batch processing" scenarios which you want to optimize overall processing time you want to prioritize throughput. Examples: 1. Erlang server: Great preemptive scheduling thus the theoretical overall computation throughput is relatively lower. But in practice, Erlang servers are extremely fast and responsive. (Prioritize latency over throughput, great for "services") 2. Traditional C/Java blocking (non-async) server spawns threads: Theoretical computation throughput is greater but in reality, it's very easy to DDoS it. (Great for batch processing but suck at "services")
advisedwang 4y ago
More accurately, this is a Latency/Throughput/Cost problem. The article assumes "max throughput" means "operate the cluster at 100% utilization". But for a parallelized service that's actually assuming they want "max throughput per $". As described, BigTable can have low latency and high throughput if you are willing to overspend on compute.
karmakaze 4y ago
Look forward to see what the follow-up post has to say. I've encountered these situations and solved it using common sense that might vary case by case. In one case we had 3 types of database servers: writer, normal-query, slow-query. The last one was added because certain queries would use up significant cpu and i/o.
PaulKeeble 4y ago
Its an obvious relationship really. In order to get to 100% cpu occupancy you either need the literally perfect spacing of every request or something is waiting somewhere. Its easier to reason about with very big tasks on a single core, 1s of CPU and tasks 0.75s long the best case is 0.75s, but if you put two tasks in there at the same time one is going to take 1.5s and actually getting to 100% CPU usage only allows for 1.3 tasks a second, the difference in latency on this very low queue behavior is a doubling in latency and queues can often get as lot longer than 1 waiting. SSDs are classic device that shows throughput increase with more queuing due to better utilisation but also substantial increases in latency from typically ms to seconds. Practically speaking the only way to really reduce latency to the minimum of the software is idle resources awaiting work.
- magicalhippo 4y ago
  > Practically speaking the only way to really reduce latency to the minimum of the software is idle resources awaiting work. Indeed. This is similar to the buffer-bloat situation, where limiting your download speed to 90-95% of max capacity will allow for both high throughput and acceptable low latency. Maybe the system could tune itself in an audio compressor[1] like attack-release style, where it would never go above say 95% utilization (random number), but could dynamically lower it if low-latency requests suffer, then slowly raising it as low-latency requests tapers off. [1]: https://themixingtips.com/what-is-compressor-attack-and-release/ https://themixingtips.com/what-is-compressor-attack-and-rele...
kasey_junk 4y ago
This skips over a solution that helps both users, make each request faster. That may seem like an obvious statement but the “solution” they propose is a complicated partitioning approach. This presumes you’ve already made each request as fast as reasonable. In my experience that’s rarely true. This also doesn’t get into the real challenge, which is the trade offs for tail latency for median (or more typically 99%) latency. Most of the time the things you do to flatten tail latency increase latency lower in the curve.
- aidenn0 4y ago
  Right, there is a performance envelope, and you only need to make the trade-off if you are already on the edge of the envelope.
notacoward 4y ago
The author's thesis largely holds for systems that are at least reasonably well implemented. It's easy to do things that hurt both latency and throughput. You can just waste lots of CPU or memory. You can add unnecessary coordination bottlenecks. You can use inefficient algorithms. Therefore, it follows that you can improve both latency and throughput by fixing these implementation details. Quite often, this will be sufficient for both latency- and throughput-sensitive users. Of particular note: even if you're already operating in a distributed, partitioned etc. environment, improving per-node efficiency often means that you can serve a given workload with fewer nodes and reduce scaling costs. In the extreme case, the less efficient system would require so many nodes even if it scaled linearly that the more-than-linear coordination costs would kill it entirely. In other words, there's always an absolute limit on how "big" your system can grow. To raise that limit you must often address both per-node performance and scaling efficiency. This is not merely hypothetical. When I was working on distributed filesystems, a competing system that was genuinely more scalable in terms of node count was so ruinously inefficient on each node that they not only couldn't beat us on latency but they basically couldn't beat us on throughput per dollar at any level either. So they resorted to politics instead of engineering, but that's another topic. The key point here is that eliminating waste will typically improve both metrics while adding scalability will only improve one at most and might actually not even raise the ceiling on that. Once a system is "well implemented" as I've described, the OP's tradeoff becomes relevant. But TBH a lot of systems just never get to that point. They end up being good enough for business purposes even when there's still tons of room for technical improvement. Often, some simple math around utilization levels and controls around batch sizes will suffice.
- kaba0 4y ago
  Good point, all too often a death-by-thousand-cuts situation arises, or a single much bigger bottleneck is the problem. As always, profiling is very very important before any form of optimization work.
hoppla 4y ago
Maybe a knee jerk reaction, but why cannot the low latency users skip the queue, or even throw out a existing job and reschedule it for later - the bulk users does not care, right? The problem reminds me how we made a lousy internet connection usable across a whole dorm using queues in m0n0wall to prioritize low latency traffic
- notacoward 4y ago
  QoS approaches like this are very well studied and very useful, but still have limits. The queuing itself consumes resources and throughput-domain requests that are already in flight will still contend with latency-domain requests (unless you want to implement preemption which is usually very difficult in a system not originally designed for it and also leads to further resource waste). If you want really low latency you have to be completely isolated from the throughput domain.
zokier 4y ago
Wouldn't a priority system enable serving both batch and low-latency requests with one cluster with good utilization? Could be furthermore improved if the batch requests can be interruptible.

25 points by vinnyglennon 4y ago

1. It's a trade-off so real-world implementations are likely to suck at both ends of the spectrum. (eg. Old Ruby servers are neither responsive nor they have large throughput) 2. The trade-off is more about computation in general. In typical "service" scenarios you almost always want to prioritize latency over throughput, while in most "batch processing" scenarios which you want to optimize overall processing time you want to prioritize throughput. Examples: 1. Erlang server: Great preemptive scheduling thus the theoretical overall computation throughput is relatively lower. But in practice, Erlang servers are extremely fast and responsive. (Prioritize latency over throughput, great for "services") 2. Traditional C/Java blocking (non-async) server spawns threads: Theoretical computation throughput is greater but in reality, it's very easy to DDoS it. (Great for batch processing but suck at "services")

advisedwang 4y ago

More accurately, this is a Latency/Throughput/Cost problem. The article assumes "max throughput" means "operate the cluster at 100% utilization". But for a parallelized service that's actually assuming they want "max throughput per $". As described, BigTable can have low latency and high throughput if you are willing to overspend on compute.

karmakaze 4y ago

Look forward to see what the follow-up post has to say. I've encountered these situations and solved it using common sense that might vary case by case. In one case we had 3 types of database servers: writer, normal-query, slow-query. The last one was added because certain queries would use up significant cpu and i/o.

PaulKeeble 4y ago

Its an obvious relationship really. In order to get to 100% cpu occupancy you either need the literally perfect spacing of every request or something is waiting somewhere. Its easier to reason about with very big tasks on a single core, 1s of CPU and tasks 0.75s long the best case is 0.75s, but if you put two tasks in there at the same time one is going to take 1.5s and actually getting to 100% CPU usage only allows for 1.3 tasks a second, the difference in latency on this very low queue behavior is a doubling in latency and queues can often get as lot longer than 1 waiting. SSDs are classic device that shows throughput increase with more queuing due to better utilisation but also substantial increases in latency from typically ms to seconds. Practically speaking the only way to really reduce latency to the minimum of the software is idle resources awaiting work.

magicalhippo 4y ago
> Practically speaking the only way to really reduce latency to the minimum of the software is idle resources awaiting work. Indeed. This is similar to the buffer-bloat situation, where limiting your download speed to 90-95% of max capacity will allow for both high throughput and acceptable low latency. Maybe the system could tune itself in an audio compressor[1] like attack-release style, where it would never go above say 95% utilization (random number), but could dynamically lower it if low-latency requests suffer, then slowly raising it as low-latency requests tapers off. [1]: https://themixingtips.com/what-is-compressor-attack-and-release/ https://themixingtips.com/what-is-compressor-attack-and-rele...

kasey_junk 4y ago

This skips over a solution that helps both users, make each request faster. That may seem like an obvious statement but the “solution” they propose is a complicated partitioning approach. This presumes you’ve already made each request as fast as reasonable. In my experience that’s rarely true. This also doesn’t get into the real challenge, which is the trade offs for tail latency for median (or more typically 99%) latency. Most of the time the things you do to flatten tail latency increase latency lower in the curve.

aidenn0 4y ago
Right, there is a performance envelope, and you only need to make the trade-off if you are already on the edge of the envelope.

notacoward 4y ago

The author's thesis largely holds for systems that are at least reasonably well implemented. It's easy to do things that hurt both latency and throughput. You can just waste lots of CPU or memory. You can add unnecessary coordination bottlenecks. You can use inefficient algorithms. Therefore, it follows that you can improve both latency and throughput by fixing these implementation details. Quite often, this will be sufficient for both latency- and throughput-sensitive users. Of particular note: even if you're already operating in a distributed, partitioned etc. environment, improving per-node efficiency often means that you can serve a given workload with fewer nodes and reduce scaling costs. In the extreme case, the less efficient system would require so many nodes even if it scaled linearly that the more-than-linear coordination costs would kill it entirely. In other words, there's always an absolute limit on how "big" your system can grow. To raise that limit you must often address both per-node performance and scaling efficiency. This is not merely hypothetical. When I was working on distributed filesystems, a competing system that was genuinely more scalable in terms of node count was so ruinously inefficient on each node that they not only couldn't beat us on latency but they basically couldn't beat us on throughput per dollar at any level either. So they resorted to politics instead of engineering, but that's another topic. The key point here is that eliminating waste will typically improve both metrics while adding scalability will only improve one at most and might actually not even raise the ceiling on that. Once a system is "well implemented" as I've described, the OP's tradeoff becomes relevant. But TBH a lot of systems just never get to that point. They end up being good enough for business purposes even when there's still tons of room for technical improvement. Often, some simple math around utilization levels and controls around batch sizes will suffice.

kaba0 4y ago
Good point, all too often a death-by-thousand-cuts situation arises, or a single much bigger bottleneck is the problem. As always, profiling is very very important before any form of optimization work.

hoppla 4y ago

Maybe a knee jerk reaction, but why cannot the low latency users skip the queue, or even throw out a existing job and reschedule it for later - the bulk users does not care, right? The problem reminds me how we made a lousy internet connection usable across a whole dorm using queues in m0n0wall to prioritize low latency traffic

notacoward 4y ago
QoS approaches like this are very well studied and very useful, but still have limits. The queuing itself consumes resources and throughput-domain requests that are already in flight will still contend with latency-domain requests (unless you want to implement preemption which is usually very difficult in a system not originally designed for it and also leads to further resource waste). If you want really low latency you have to be completely isolated from the throughput domain.

zokier 4y ago

Wouldn't a priority system enable serving both batch and low-latency requests with one cluster with good utilization? Could be furthermore improved if the batch requests can be interruptible.