3 ms·

I think it worth noting briefly that almost everything discussed in TFA concerns bandwidth (network, disk, other I/O, CPU) and not latency. That's understandabl

I think it worth noting briefly that almost everything discussed in TFA concerns bandwidth (network, disk, other I/O, CPU) and not latency. That's understandable, because performance for a lot of people is about bandwidth. But there are some of us for whom bandwidth definitely takes a back seat, and you need a different set of tools for tuning latency in Linux.

meisel 3y ago
What are the tools and resources you would recommend with respect to latency tuning?
- klysm 3y ago
  Latency of what particular system?
  - SkyMarshal 3y ago
    I'm sure there's lots of info on tuning linux servers, both for throughput and latency, so how about for desktop workstation use cases?
- interroboink 3y ago
  Not the person you asked, but generally you might want to look at "frame-based" profilers. These are typically used in video games, but the concept is general, and can apply to other applications. The "frame" could also be something like a request or transaction being processed. I like Tracy[1], myself. Another latency metric that you'll see, often w/respect to web apps and microservices is "P99" and similar. This is the amount of time in which 99% of requests get their response. For a higher percentile, you get a better idea of worst-case performance. [1] https://github.com/wolfpld/tracy https://github.com/wolfpld/tracy
dan-robertson 3y ago
That’s a pretty reasonable point. It’s much easier to increase bandwidth than latency too[1] so often caring about latency can be important. 0. Another thing people may want to optimise for is performance per watt, but I won’t say much more about it. 1. There are cases where bandwidth optimisations are latency optimisations, eg if you can fit more of your processes onto one box, you are reducing the average distance between the processes and whatever they talk to and hence average latency 2. A very obvious thing to do when optimising for latency is increase bandwidth enough that the bandwidth doesn’t throttle you 3. I feel like mostly if you are aggressively optimising latency, there isn’t much Linux tuning to do. Maybe I’m wrong – I don’t really know much about this – but I think it’s mostly pinning to a core, running tickles, doing user space networking, and then hardwarey things like tuning page size, SMT, power-saving settings, and other things like choice of hardware. [1] https://pdfs.semanticscholar.org/bce7/5f78d340cac32dccd8631f59eedebf192abe.pdf https://pdfs.semanticscholar.org/bce7/5f78d340cac32dccd8631f...
stefan_ 3y ago
With the caution that if your goal is to reduce latency for the realtime properties of your system, chances are that you can turn any number of different knobs on your out of the box Linux distribution but it will not result in a satisfactory system.
toast0 3y ago
I would say throughput rather than bandwidth, although the meaning is similar, people have been discussing throughput vs latency for a long time. Sometimes you can improve both, but often it's a tradeoff.

meisel 3y ago

What are the tools and resources you would recommend with respect to latency tuning?

klysm 3y ago
Latency of what particular system?
- SkyMarshal 3y ago
  I'm sure there's lots of info on tuning linux servers, both for throughput and latency, so how about for desktop workstation use cases?
interroboink 3y ago
Not the person you asked, but generally you might want to look at "frame-based" profilers. These are typically used in video games, but the concept is general, and can apply to other applications. The "frame" could also be something like a request or transaction being processed. I like Tracy[1], myself. Another latency metric that you'll see, often w/respect to web apps and microservices is "P99" and similar. This is the amount of time in which 99% of requests get their response. For a higher percentile, you get a better idea of worst-case performance. [1] https://github.com/wolfpld/tracy https://github.com/wolfpld/tracy

dan-robertson 3y ago

That’s a pretty reasonable point. It’s much easier to increase bandwidth than latency too[1] so often caring about latency can be important. 0. Another thing people may want to optimise for is performance per watt, but I won’t say much more about it. 1. There are cases where bandwidth optimisations are latency optimisations, eg if you can fit more of your processes onto one box, you are reducing the average distance between the processes and whatever they talk to and hence average latency 2. A very obvious thing to do when optimising for latency is increase bandwidth enough that the bandwidth doesn’t throttle you 3. I feel like mostly if you are aggressively optimising latency, there isn’t much Linux tuning to do. Maybe I’m wrong – I don’t really know much about this – but I think it’s mostly pinning to a core, running tickles, doing user space networking, and then hardwarey things like tuning page size, SMT, power-saving settings, and other things like choice of hardware. [1] https://pdfs.semanticscholar.org/bce7/5f78d340cac32dccd8631f59eedebf192abe.pdf https://pdfs.semanticscholar.org/bce7/5f78d340cac32dccd8631f...

stefan_ 3y ago

With the caution that if your goal is to reduce latency for the realtime properties of your system, chances are that you can turn any number of different knobs on your out of the box Linux distribution but it will not result in a satisfactory system.

toast0 3y ago

I would say throughput rather than bandwidth, although the meaning is similar, people have been discussing throughput vs latency for a long time. Sometimes you can improve both, but often it's a tradeoff.