5 ms·

Ubuntu Looking at Applying Low-Latency Optimizations to Its Generic Kernel

126 points by mfiguiere 3y ago

bitcharmer 3y ago
This is a welcome change. Even though I moved away from Ubuntu and onto Fedora, I haven't used generic kernel in years on my desktop. Low latency variant with low-tick scheduling (I hate the false term "tickless"), rcu offloading and higher scheduling hr tick frequency makes a noticeable difference in responsiveness. I hope other major distros follow suit.
- slekker 3y ago
  Why aren't these changes the default? What are their drawbacks?
  - Tuna-Fish 3y ago
    Generally, worse performance on throughput loads.
  - michaelt 3y ago
    When a single-CPU system performs multi-tasking, switching between tasks is known as 'context switching' https://en.wikipedia.org/wiki/Context_switch https://en.wikipedia.org/wiki/Context_switch and it comes with a time cost. For example, the old process has its data in the CPU's fastest on-die cache, the new one doesn't - so it's initially slower. And the same thing can happen when switching back. For the lowest latency applications, there's also competition from "just dedicate a core to it full time" and "offload it to a dedicated microcontroller" lowering the demand to get it merged.
    - pzmarzly 3y ago
      > just dedicate a core to it full time That's interesting, is there a place to read more about it? E.g. how to set it up on Linux and how much does it help with audio.
      - michaelt 3y ago
        I've never done this myself, but as I understand it [1] you can choose certain cores that the OS scheduler doesn't schedule anything on, then you can manually assign processes to them. Assigning an entire CPU core to audio might be a bit much, unless you've got CPU cores to spare - the context in which I heard of this was high frequency trading, where the server really didn't need to do much except run this one application. [1] https://unix.stackexchange.com/a/530579 https://unix.stackexchange.com/a/530579
        Joeboy 3y ago
        I'd have thought most people doing serious audio work (recording live performances etc) would have a core to spare for the audio process. I suppose there's probably some reason this doesn't work well, because I've never seen anybody suggest it for linux audio. Or maybe it's just that low latency was "solved" by pre-emption or whatever when cores were a scarcer resource.
        hermitdev 3y ago
        Things to search for: CPU sets, core isolation. I work in trading, pretty much all of our performance sensitive apps use core isolation.
        dijit 3y ago
        same is true of gameservers for the serious AAA studios. On windows this sort of comes out of the box because windows always defaults CORE0 to the OS and is mostly single threaded, but you can set CPU affinity on your processes in process: just avoid core0. https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-setprocessaffinitymask https://learn.microsoft.com/en-us/windows/win32/api/winbase/...
        csdvrx 3y ago
        I love windows, so when I moved to Linux I tried to replicate this on my Linux box I've found a lot of drivers like to spread their interrupts all over the place, so it's not as simple as using cpu affinity (or pinning processes) outside core 0 :(
freedomben 3y ago
This is really great news: > Ubuntu's low-latency kernel is mostly Kconfig configuration changes applied to their kernel build that they are now at least considering to make by default. Ubuntu used to have an absolute monster of a hacked up franken-kernel compared to mainstream, but (very fortunately) over the years it's gotten much, much closer to upstream. It was honestly a bit of a nightmare trying to apply a set of custom patches on top of it (to improve a driver that was buggy for many years that I hacked in a way that made it work awesome but would never be accepted upstream because it violated FCC regulations). I'm an outside observer so pure speculation, but I have suspected that it was painful for the Canonical kernel team just as it was for me, and that they made a concerted effort to get away from the craziness. In related news, the people who backport kernel fixes to LTS versions they maintain for their distros (think Canonical and Red Hat, for example) are some of the most unsung heroes in the world. That work is brutal and takes a lot of dedication, and when you're doing your job perfectly nobody knows, but when you mess up everybody knows. That leads to a lot of thanklessness.
- csdvrx 3y ago
  > to improve a driver that was buggy for many years that I hacked in a way that made it work awesome but would never be accepted upstream because it violated FCC regulations Now I want to know more! Was it a misbehaving wifi? Did it violate the spirit, or the letter of the regulations? I've had the opposite problem, trying to make a card respect FCC regulations (ieee80211_regdom=US) while its firmware seems to have another country hardcoded. I think it would be far better to use regdom=US instead of letting the card do whatever's ok in the country the card believes it's still operating in, yet it would also "violate" FCC regulations to let me write into its firmware that it should follow US rules by default from now on. ebay + a worldwide market + "can't trust people to change the firmware" = problems I wish there was more flexibility in the drivers to correct such misbehaviors, because I believe most people want to do the right things.
- crote 3y ago
  Why would violating FCC regulations prevent it from being merged? There are plenty of Linux users who do not live within the jurisdiction of the FCC. Similarly, there are many regions which forbid certain spectrum use which the FCC allows. Surely it'd make way more sense to just have a locale configuration option, rather than only allowing the intersection of all radio emission rules worldwide?
dvh 3y ago
There was an Australian anesthesiologist (whose name escapes me atm) and he did lot of work on kernel latency but I think he gave up.
- renewiltord 3y ago
  https://en.m.wikipedia.org/wiki/Con_Kolivas https://en.m.wikipedia.org/wiki/Con_Kolivas The CK patches were always something that many of us eagerly applied at the time.
- fransje26 3y ago
  Not that long ago I read that he was back in the game. If memory serves me, his domain of interest was kernel task scheduling and fairness.
- ahartmetz 3y ago
  At least he raised awareness, and things are in somewhat better shape now with the BFQ I/O scheduler (mostly obsoleted by SSDs though...) and EEVDF CPU scheduler. I care about latency and I use the "lowlatency" flavor of the Ubuntu kernel. It's nice.
- matheusmoreira 3y ago
  An anesthesiologist working on the kernel? That's awesome and a huge inspiration for me.
a-french-anon 3y ago
>CONFIG_NO_HZ_FULL for full tickless mode Huh? As a Gentoo user who searched about all of this, once upon a time, I remember this being worse for latency, introduced as an Android patch for battery life gains. Also, what's the point of setting CONFIG_HZ=1000 with it?
- csdvrx 3y ago
  > what's the point of setting CONFIG_HZ=1000 with it? Agreed, if you go "tickless", you want to go to the lowest value for what's still ticking: I have my own kernel configs, and I generally use HZ=100: # zcat /proc/config.gz |grep HZ_ CONFIG_NO_HZ_COMMON=y # CONFIG_HZ_PERIODIC is not set # CONFIG_NO_HZ_IDLE is not set CONFIG_NO_HZ_FULL=y CONFIG_HZ_100=y # CONFIG_HZ_250 is not set # CONFIG_HZ_300 is not set # CONFIG_HZ_1000 is not set CONFIG_MACHZ_WDT=m For Little.Big heterogeneous cores (like on the i7-1270P Alder Lake P), I suggest also thinking about how the cores are arranged to group the interrupts intelligently, and not just go by which cores are efficiency/power as /usr/local/bin/coretype simply reports: P CORES: 0..7 E Cores: 8..19 You should check which cores are on a core.id with: cat /proc/cpuinfo |egrep "physical|core.id|cache.size|processor" |grep -E "core.id|processor" On my cpu: processor : 0 core id : 0 processor : 1 core id : 0 processor : 2 core id : 4 processor : 3 core id : 4 processor : 4 core id : 8 processor : 5 (....) Therefore you could want to: - Leave efficiency cores 8..19 as-is (nohz_full is not ideal and consumes power, leave the efficiency cores alone - they are made to be efficient!) - Use power core 0 as normal - Put all the other power cores 1-7 but 4 in NOHZ_FULL - Put all IRQ and callbacks on power core cpu 4: for performance + a race to sleep So on my i7-1270P, I use: `nohz_full=1-3,5-7 rcu_nocbs=0-3,5-7 irqaffinity=4` In theory, you get the best of both worlds. In practice, unfortunately, it's not perfect as the NVMe and WIFI drivers require extra care or they'll sprinkle interrupts everywhere which you can check with just cat /proc/interrupts For NVMe, you can limit the queues for nocbs with `nvme.poll_queues=1 nvme.write_queues=1` For wifi, you can try to limit its eagerness to have more interrupts with a simple script like `for irq in $(grep -v " ...: ......... 0 0 0 ......... 0 0 0 0 0 0 0 0 0 0 0" /proc/interrupts |sed -e 's/:.//g' -e 's/^ //g' -e 's/ .//g' |grep -v '^$') do [ -f /proc/irq/$irq/smp_affinity_list ] \ && echo 4 > /proc/irq/$irq/smp_affinity_list \ || echo skipping $irq done` BTW, if you have more IO and you prefer having interrupts spread a little more, you can use both cores one of the avx512-less efficiency core.id 8 with irqaffinity=4-5 Then the cmdline becomes: cpu0_hotplug nohz_full=1-3,5-7 nr_running=1 rcu_nocbs=0-3,6-7 irqaffinity=4-5 EDIT: I added some clarifications, it's very late but I realized it required a little more context
cwillu 3y ago
Note that in spite of what is said, this is not remotely equivalent to the true rt patchset for low latency. They switched ubuntu studio to the non-rt “lowlatency” and all of a sudden I have to increase my audio buffers by about 10x to get jack to run xrun free, which leads to some pretty horrific latency on a midi keyboard. I can easily run buffer sizes for 0.5ms latency on the rt patchset while running random twitch streams and games at the same time, but struggle to maintain 10ms on lowlatency without xruns with everything but the core audio apps shut down.
- stinos 3y ago
  Are there distros for which this is fixed out of the box already? About 20 years ago we were writing a cross-platform application which required relaltively low latencies (2-5mSec IIRC) for audio with some of the RME interfaces. On Windows (with ASIO) and MacOS (don't remember) this basically just worked and we'd only have to take care not to run too much other heavy bandwith applications at the same time. On any of the *nixes out there it was completely hit or miss. Even after the struggle to get the correct drivers working, it required a myriad of seemingly obscure tweaks and kernel builds only to result in a system which was usually rather unstable at best. Plus the realization that it would be pretty much impossible to ship this because it would require the other party to also apply similar tweaks specific to their distro. I would have thought 20 years would be enough to mature this especially in major distros like Ubuntu but appaprently that is not the case?
  - cwillu 3y ago
    The only issue I ever have any more is the scheduler latency for the actual running software; the drivers have generally worked fine for all sorts of random (but class-compliant) devices.
  - jauntywundrkind 3y ago
    Wow that sounds awful. I've had no trouble plugging in random PCIe or USB devices and getting <6ms buffers. Everything works fine. After some years deeply enjoying JACK's incredible flexibility & latency, I eventually switched to PulseAudio with mostly stock config but much reduced buffers. Now a days PipeWire is the best of both worlds. It'd be interesting to see if the top post still had such issues as they do under PipeWire, although I'm unsure what about pipewires scheduling would have better mechanistic sympathy. Plenty of distros do offer -rt kernels. But -rt has been a massive set of patches for a long long time, and it's a different operating mode, and it has tradeoffs. Extreme latency sensitivity is a niche need, and giving it to everyone would have negative impacts. That's why it's not the default. Good news though, -rt's longkng effort to upstream appears to be nearing a successful conclusion, so realtime may be a Kconfig away in the future. We just have to solve printk() first, ha! https://lwn.net/Articles/951337/ https://lwn.net/Articles/951337/
justaj 3y ago
I don't really understand. I thought those patches were already merged into mainline a couple of years ago. For example you can have `CONFIG_PREEMPT=y` and use linux-lowlatency kernel (which AFAIK is just the vanilla kernel with special configuration)
- lathiat 3y ago
  Try read the opening paragraph of the article :)
b112 3y ago
Probably a priority, as everything Ubuntu is on laggy, oversized snap.
Kalanos 3y ago
what are the implications? does that speed unlock new use cases like new apps or gaming or is stuff just going to be slightly faster?

126 points by mfiguiere 3y ago

This is a welcome change. Even though I moved away from Ubuntu and onto Fedora, I haven't used generic kernel in years on my desktop. Low latency variant with low-tick scheduling (I hate the false term "tickless"), rcu offloading and higher scheduling hr tick frequency makes a noticeable difference in responsiveness. I hope other major distros follow suit.

slekker 3y ago
Why aren't these changes the default? What are their drawbacks?
- Tuna-Fish 3y ago
  Generally, worse performance on throughput loads.
- michaelt 3y ago
  When a single-CPU system performs multi-tasking, switching between tasks is known as 'context switching' https://en.wikipedia.org/wiki/Context_switch https://en.wikipedia.org/wiki/Context_switch and it comes with a time cost. For example, the old process has its data in the CPU's fastest on-die cache, the new one doesn't - so it's initially slower. And the same thing can happen when switching back. For the lowest latency applications, there's also competition from "just dedicate a core to it full time" and "offload it to a dedicated microcontroller" lowering the demand to get it merged.
  - pzmarzly 3y ago
    > just dedicate a core to it full time That's interesting, is there a place to read more about it? E.g. how to set it up on Linux and how much does it help with audio.
    - michaelt 3y ago
      I've never done this myself, but as I understand it [1] you can choose certain cores that the OS scheduler doesn't schedule anything on, then you can manually assign processes to them. Assigning an entire CPU core to audio might be a bit much, unless you've got CPU cores to spare - the context in which I heard of this was high frequency trading, where the server really didn't need to do much except run this one application. [1] https://unix.stackexchange.com/a/530579 https://unix.stackexchange.com/a/530579
      - Joeboy 3y ago
        I'd have thought most people doing serious audio work (recording live performances etc) would have a core to spare for the audio process. I suppose there's probably some reason this doesn't work well, because I've never seen anybody suggest it for linux audio. Or maybe it's just that low latency was "solved" by pre-emption or whatever when cores were a scarcer resource.
      - hermitdev 3y ago
        Things to search for: CPU sets, core isolation. I work in trading, pretty much all of our performance sensitive apps use core isolation.
        dijit 3y ago
        same is true of gameservers for the serious AAA studios. On windows this sort of comes out of the box because windows always defaults CORE0 to the OS and is mostly single threaded, but you can set CPU affinity on your processes in process: just avoid core0. https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-setprocessaffinitymask https://learn.microsoft.com/en-us/windows/win32/api/winbase/...
        csdvrx 3y ago
        I love windows, so when I moved to Linux I tried to replicate this on my Linux box I've found a lot of drivers like to spread their interrupts all over the place, so it's not as simple as using cpu affinity (or pinning processes) outside core 0 :(

freedomben 3y ago

This is really great news: > Ubuntu's low-latency kernel is mostly Kconfig configuration changes applied to their kernel build that they are now at least considering to make by default. Ubuntu used to have an absolute monster of a hacked up franken-kernel compared to mainstream, but (very fortunately) over the years it's gotten much, much closer to upstream. It was honestly a bit of a nightmare trying to apply a set of custom patches on top of it (to improve a driver that was buggy for many years that I hacked in a way that made it work awesome but would never be accepted upstream because it violated FCC regulations). I'm an outside observer so pure speculation, but I have suspected that it was painful for the Canonical kernel team just as it was for me, and that they made a concerted effort to get away from the craziness. In related news, the people who backport kernel fixes to LTS versions they maintain for their distros (think Canonical and Red Hat, for example) are some of the most unsung heroes in the world. That work is brutal and takes a lot of dedication, and when you're doing your job perfectly nobody knows, but when you mess up everybody knows. That leads to a lot of thanklessness.

csdvrx 3y ago
> to improve a driver that was buggy for many years that I hacked in a way that made it work awesome but would never be accepted upstream because it violated FCC regulations Now I want to know more! Was it a misbehaving wifi? Did it violate the spirit, or the letter of the regulations? I've had the opposite problem, trying to make a card respect FCC regulations (ieee80211_regdom=US) while its firmware seems to have another country hardcoded. I think it would be far better to use regdom=US instead of letting the card do whatever's ok in the country the card believes it's still operating in, yet it would also "violate" FCC regulations to let me write into its firmware that it should follow US rules by default from now on. ebay + a worldwide market + "can't trust people to change the firmware" = problems I wish there was more flexibility in the drivers to correct such misbehaviors, because I believe most people want to do the right things.
crote 3y ago
Why would violating FCC regulations prevent it from being merged? There are plenty of Linux users who do not live within the jurisdiction of the FCC. Similarly, there are many regions which forbid certain spectrum use which the FCC allows. Surely it'd make way more sense to just have a locale configuration option, rather than only allowing the intersection of all radio emission rules worldwide?

dvh 3y ago

There was an Australian anesthesiologist (whose name escapes me atm) and he did lot of work on kernel latency but I think he gave up.

renewiltord 3y ago
https://en.m.wikipedia.org/wiki/Con_Kolivas https://en.m.wikipedia.org/wiki/Con_Kolivas The CK patches were always something that many of us eagerly applied at the time.
fransje26 3y ago
Not that long ago I read that he was back in the game. If memory serves me, his domain of interest was kernel task scheduling and fairness.
ahartmetz 3y ago
At least he raised awareness, and things are in somewhat better shape now with the BFQ I/O scheduler (mostly obsoleted by SSDs though...) and EEVDF CPU scheduler. I care about latency and I use the "lowlatency" flavor of the Ubuntu kernel. It's nice.
matheusmoreira 3y ago
An anesthesiologist working on the kernel? That's awesome and a huge inspiration for me.

a-french-anon 3y ago

>CONFIG_NO_HZ_FULL for full tickless mode Huh? As a Gentoo user who searched about all of this, once upon a time, I remember this being worse for latency, introduced as an Android patch for battery life gains. Also, what's the point of setting CONFIG_HZ=1000 with it?

csdvrx 3y ago
> what's the point of setting CONFIG_HZ=1000 with it? Agreed, if you go "tickless", you want to go to the lowest value for what's still ticking: I have my own kernel configs, and I generally use HZ=100: # zcat /proc/config.gz |grep HZ_ CONFIG_NO_HZ_COMMON=y # CONFIG_HZ_PERIODIC is not set # CONFIG_NO_HZ_IDLE is not set CONFIG_NO_HZ_FULL=y CONFIG_HZ_100=y # CONFIG_HZ_250 is not set # CONFIG_HZ_300 is not set # CONFIG_HZ_1000 is not set CONFIG_MACHZ_WDT=m For Little.Big heterogeneous cores (like on the i7-1270P Alder Lake P), I suggest also thinking about how the cores are arranged to group the interrupts intelligently, and not just go by which cores are efficiency/power as /usr/local/bin/coretype simply reports: P CORES: 0..7 E Cores: 8..19 You should check which cores are on a core.id with: cat /proc/cpuinfo |egrep "physical|core.id|cache.size|processor" |grep -E "core.id|processor" On my cpu: processor : 0 core id : 0 processor : 1 core id : 0 processor : 2 core id : 4 processor : 3 core id : 4 processor : 4 core id : 8 processor : 5 (....) Therefore you could want to: - Leave efficiency cores 8..19 as-is (nohz_full is not ideal and consumes power, leave the efficiency cores alone - they are made to be efficient!) - Use power core 0 as normal - Put all the other power cores 1-7 but 4 in NOHZ_FULL - Put all IRQ and callbacks on power core cpu 4: for performance + a race to sleep So on my i7-1270P, I use: `nohz_full=1-3,5-7 rcu_nocbs=0-3,5-7 irqaffinity=4` In theory, you get the best of both worlds. In practice, unfortunately, it's not perfect as the NVMe and WIFI drivers require extra care or they'll sprinkle interrupts everywhere which you can check with just cat /proc/interrupts For NVMe, you can limit the queues for nocbs with `nvme.poll_queues=1 nvme.write_queues=1` For wifi, you can try to limit its eagerness to have more interrupts with a simple script like `for irq in $(grep -v " ...: ......... 0 0 0 ......... 0 0 0 0 0 0 0 0 0 0 0" /proc/interrupts |sed -e 's/:.//g' -e 's/^ //g' -e 's/ .//g' |grep -v '^$') do [ -f /proc/irq/$irq/smp_affinity_list ] \ && echo 4 > /proc/irq/$irq/smp_affinity_list \ || echo skipping $irq done` BTW, if you have more IO and you prefer having interrupts spread a little more, you can use both cores one of the avx512-less efficiency core.id 8 with irqaffinity=4-5 Then the cmdline becomes: cpu0_hotplug nohz_full=1-3,5-7 nr_running=1 rcu_nocbs=0-3,6-7 irqaffinity=4-5 EDIT: I added some clarifications, it's very late but I realized it required a little more context

cwillu 3y ago

Note that in spite of what is said, this is not remotely equivalent to the true rt patchset for low latency. They switched ubuntu studio to the non-rt “lowlatency” and all of a sudden I have to increase my audio buffers by about 10x to get jack to run xrun free, which leads to some pretty horrific latency on a midi keyboard. I can easily run buffer sizes for 0.5ms latency on the rt patchset while running random twitch streams and games at the same time, but struggle to maintain 10ms on lowlatency without xruns with everything but the core audio apps shut down.

stinos 3y ago
Are there distros for which this is fixed out of the box already? About 20 years ago we were writing a cross-platform application which required relaltively low latencies (2-5mSec IIRC) for audio with some of the RME interfaces. On Windows (with ASIO) and MacOS (don't remember) this basically just worked and we'd only have to take care not to run too much other heavy bandwith applications at the same time. On any of the *nixes out there it was completely hit or miss. Even after the struggle to get the correct drivers working, it required a myriad of seemingly obscure tweaks and kernel builds only to result in a system which was usually rather unstable at best. Plus the realization that it would be pretty much impossible to ship this because it would require the other party to also apply similar tweaks specific to their distro. I would have thought 20 years would be enough to mature this especially in major distros like Ubuntu but appaprently that is not the case?
- cwillu 3y ago
  The only issue I ever have any more is the scheduler latency for the actual running software; the drivers have generally worked fine for all sorts of random (but class-compliant) devices.
- jauntywundrkind 3y ago
  Wow that sounds awful. I've had no trouble plugging in random PCIe or USB devices and getting <6ms buffers. Everything works fine. After some years deeply enjoying JACK's incredible flexibility & latency, I eventually switched to PulseAudio with mostly stock config but much reduced buffers. Now a days PipeWire is the best of both worlds. It'd be interesting to see if the top post still had such issues as they do under PipeWire, although I'm unsure what about pipewires scheduling would have better mechanistic sympathy. Plenty of distros do offer -rt kernels. But -rt has been a massive set of patches for a long long time, and it's a different operating mode, and it has tradeoffs. Extreme latency sensitivity is a niche need, and giving it to everyone would have negative impacts. That's why it's not the default. Good news though, -rt's longkng effort to upstream appears to be nearing a successful conclusion, so realtime may be a Kconfig away in the future. We just have to solve printk() first, ha! https://lwn.net/Articles/951337/ https://lwn.net/Articles/951337/

justaj 3y ago

I don't really understand. I thought those patches were already merged into mainline a couple of years ago. For example you can have `CONFIG_PREEMPT=y` and use linux-lowlatency kernel (which AFAIK is just the vanilla kernel with special configuration)

lathiat 3y ago
Try read the opening paragraph of the article :)

b112 3y ago

Probably a priority, as everything Ubuntu is on laggy, oversized snap.

Kalanos 3y ago

what are the implications? does that speed unlock new use cases like new apps or gaming or is stuff just going to be slightly faster?