4 ms·

Its an obvious relationship really. In order to get to 100% cpu occupancy you either need the literally perfect spacing of every request or something is waiting

Its an obvious relationship really. In order to get to 100% cpu occupancy you either need the literally perfect spacing of every request or something is waiting somewhere. Its easier to reason about with very big tasks on a single core, 1s of CPU and tasks 0.75s long the best case is 0.75s, but if you put two tasks in there at the same time one is going to take 1.5s and actually getting to 100% CPU usage only allows for 1.3 tasks a second, the difference in latency on this very low queue behavior is a doubling in latency and queues can often get as lot longer than 1 waiting. SSDs are classic device that shows throughput increase with more queuing due to better utilisation but also substantial increases in latency from typically ms to seconds. Practically speaking the only way to really reduce latency to the minimum of the software is idle resources awaiting work.

magicalhippo 4y ago
> Practically speaking the only way to really reduce latency to the minimum of the software is idle resources awaiting work. Indeed. This is similar to the buffer-bloat situation, where limiting your download speed to 90-95% of max capacity will allow for both high throughput and acceptable low latency. Maybe the system could tune itself in an audio compressor[1] like attack-release style, where it would never go above say 95% utilization (random number), but could dynamically lower it if low-latency requests suffer, then slowly raising it as low-latency requests tapers off. [1]: https://themixingtips.com/what-is-compressor-attack-and-release/ https://themixingtips.com/what-is-compressor-attack-and-rele...