6 ms·

Will this eventually get merged into the main Linux branch? Are there any downsides to using the real time kernel for more mainstream tasks?

capableweb 5y ago
> Are there any downsides to using the real time kernel for more mainstream tasks? Real-time kernels tend to sacrifice anything in order to minimize latency, for being better for use cases where lowest latency is needed. This usually means tradeoffs regarding throughput, power consumption, resource usage, reliability and more, in order to get the lowest possible latency. In some cases you need really low latency, but in most day-to-day tasks I don't think you need it (disclaimer: I run linux-zen personally, mainly for minimizing audio latency when doing music with bunch of hardware + desktop PC). > Will this eventually get merged into the main Linux branch? I'm not too familiar with kernel development so someone else should give you a definitive answer, but considering the tradeoffs it's very unlikely.
- deleted 5y ago
  [deleted]
- R0b0t1 5y ago
  Do you have concrete examples of any of those things? The only real tradeoff I am aware of is you sacrifice IO throughput so that the kernel can spread time related service tasks more fairly and userland process can interrupt the kernel. But, as modern Linux kernels work, it is not obvious you experience these throughput problems if you don't mark any tasks with realtime priority. You can still get low power consumption, low resource usage, high reliability, etc, with realtime requirements.
  - sudorook 5y ago
    The degree to which the OS deviates from the vanilla kernel behavior depends on the requirements of the processes being run in real-time. For example, to run hard real-time, high frequency (>10 kHz) tasks with low latency and jitter, one needs to disable CPU frequency scaling (among other things) when compiling the kernel. The result is higher power consumption.
    - R0b0t1 5y ago
      You don't have to do those things, it's just easiest to sometimes. It depends entirely on your hardware and your realtime requirements. If you do in-code synchronization you'll usually have problems with throttling but if you can synchronize against a stable timer/counter you will have fewer issues. It's also worth noting that some of these tasks people throw out are just not a very good fit for even a real time kernel. The main use on Linux seems to be sound and video, which uses specialized hardware. The specialized hardware for other operations is typically not available on Linux capable SoCs and it's worth communicating with a different chip to do those things. The premise of some of these tradeoffs is not always valid.
    - EE84M3i 5y ago
      Forgive my ignorance, but could you explain a bit more why frequency scaling needs to be disabled at kernel compile time, and can't be disabled at runtime via sysctl?
      - sudorook 5y ago
        I elided a lot detail for the sake of making a simple example about power consumption. For applications that don't need hard real-time and/or aren't very high frequency, one could set the CPU frequency governor to always run at max frequency. You're right in that this can be done at runtime without a need to disable CONFIG_CPU_FREQ. To achieve hard real-time at high frequency (roughly >10kHz), the standard kernel and PREEMPT_RT are not sufficient. One approach is to use a microkernel that preempts the Linux kernel. RTAI and Xenomai are two frameworks that do this. Processes run through them cannot be preempted by the kernel, so deterministic timings can be much better guaranteed. It's in this case that one would disable CPU scaling (along with Intel pstates, ACPI power management, and other things I don't remember offhand) when compiling the kernel. Changes in CPU frequency can change the frequency of the time counters on the CPU, and that opens up a vector for the vanilla kernel to interfere with the timers used by the microkernel and potentially break real-time. Now, is it true in this latter scenario that one, as I said, "needs to" disable CPU scaling entirely? Now that you've got me thinking about it, probably not. If one guarantees that the 'performance' (i.e. max freq) governor is running at boot, leaving CONFIG_CPU_FREQ enabled may be fine. (The default governor will still need to be changed to the performance one in the kernel config, though.) Setting max frequency at runtime, however, maybe not. This is probably an edge case (I haven't tested this), but if the system boots with a dynamic CPU frequency enabled, the standard kernel, which cannot access information about threads controlled by the microkernel, could scale down the CPU a real-time thread is running on and mess up the timings. Changing the frequency mode at runtime after that happens may not fix that issue. If I were to write that again, I'd use "should" or instead of "needs to." It's much simpler to disable frequency scaling in the menuconfig while disabling everything else than it is to leave it in and worry about it afterward. None of this changes the point about real-time and power consumption, though.
- borman 5y ago
  > where lowest latency is needed Not so sure it would even be "the lowest", rather than "low enough, stable and predictable", even if it's higher on average than optimal. For example, comparing a hashtable lookup to a BST lookup (and ignoring the memory hierarchy for the sake of example), the former would be faster on average, but the latter would produce more predictable lookup times without hiccups.
- duped 5y ago
  Slight nit, real time systems are designed for a maximum worst case latency within some spec, not minimum latency. You can make a system that has a lower best case latency but that doesn't matter if the worst case causes a missed deadline. And in the context of real time systems, a missed deadline is either equivalent to a catastrophic error (hard real time) or really terrible but non catastrophic error (soft real time). Soft real time applications like audio (no one dies because of a buffer underrun) can sacrifice the maximum worst case for better average case, because the worst case depends on load and can be worked around. Hard real time cares a lot more about the maximum worst case, like say a CNC machine where missing a deadline can cause a mechanical failure and break the part.
  - chippiewill 5y ago
    > a missed deadline is either equivalent to a catastrophic error (hard real time) or really terrible but non catastrophic error (soft real time) I think you're conflating soft/hard real time with criticality (although they're often aligned) Hard real time refers to a total system failure, even if it has trivial consequences. For example if a printer has a timing error at any point the page it's printing is irrecoverably ruined. It's hard real-time but low criticality. Soft real time is where you can miss deadlines but the performance / experience degrades rather than fails. e.g. frame rate drops in a video game. The frames are still valid when delivered late, but the experience is poor. > Slight nit, real time systems are designed for a maximum worst case latency within some spec, not minimum latency. Indeed, in fact at times you don't even want a substantially lower minimum latency because it may cause unreasonable jitter in your output.
- chippiewill 5y ago
  > I'm not too familiar with kernel development so someone else should give you a definitive answer, but considering the tradeoffs it's very unlikely. I believe most of PREEMPT_RT has already been merged over the years. The expensive bits are all guarded behind a compile time flag.
marcan_42 5y ago
It's already being merged; PREEMPT_RT got merged last year. We're close to the endgame on that front. There are some performance tradeoffs to using an RT kernel; you trade some throughput for determinism. That said, I've been running RT on my desktop most of the time for years and never had any issues.
- pabs3 5y ago
  After all the patches are merged, will there be work on making PREEMPT_RT a boot-time or run-time choice rather than build-time? Why did you choose RT for your desktop?
  - op00to 5y ago
    Id consider RT if I was serious about audio.
  - marcan_42 5y ago
    I do audio stuff :)
    - justaj 5y ago
      And luckily _much_ more! (I'm deeply grateful for your contributions btw)
  - chippiewill 5y ago
    > will there be work on making PREEMPT_RT a boot-time or run-time choice rather than build-time? Given the performance critical nature of how this changes things I personally think it's unlikely. It does stuff like change the way spinlocks work in the Kernel which would be not only difficult, but expensive without recompilation. The best I think you'll get is a boot-time option to switch between pre-compiled kernels. That being said, most of the RT Linux patches over the years are actually compiled in all the time anyway so a lot of these improvements have been incorporated for a long time.
    - 5y ago

capableweb 5y ago

> Are there any downsides to using the real time kernel for more mainstream tasks? Real-time kernels tend to sacrifice anything in order to minimize latency, for being better for use cases where lowest latency is needed. This usually means tradeoffs regarding throughput, power consumption, resource usage, reliability and more, in order to get the lowest possible latency. In some cases you need really low latency, but in most day-to-day tasks I don't think you need it (disclaimer: I run linux-zen personally, mainly for minimizing audio latency when doing music with bunch of hardware + desktop PC). > Will this eventually get merged into the main Linux branch? I'm not too familiar with kernel development so someone else should give you a definitive answer, but considering the tradeoffs it's very unlikely.

deleted 5y ago
[deleted]
R0b0t1 5y ago
Do you have concrete examples of any of those things? The only real tradeoff I am aware of is you sacrifice IO throughput so that the kernel can spread time related service tasks more fairly and userland process can interrupt the kernel. But, as modern Linux kernels work, it is not obvious you experience these throughput problems if you don't mark any tasks with realtime priority. You can still get low power consumption, low resource usage, high reliability, etc, with realtime requirements.
- sudorook 5y ago
  The degree to which the OS deviates from the vanilla kernel behavior depends on the requirements of the processes being run in real-time. For example, to run hard real-time, high frequency (>10 kHz) tasks with low latency and jitter, one needs to disable CPU frequency scaling (among other things) when compiling the kernel. The result is higher power consumption.
  - R0b0t1 5y ago
    You don't have to do those things, it's just easiest to sometimes. It depends entirely on your hardware and your realtime requirements. If you do in-code synchronization you'll usually have problems with throttling but if you can synchronize against a stable timer/counter you will have fewer issues. It's also worth noting that some of these tasks people throw out are just not a very good fit for even a real time kernel. The main use on Linux seems to be sound and video, which uses specialized hardware. The specialized hardware for other operations is typically not available on Linux capable SoCs and it's worth communicating with a different chip to do those things. The premise of some of these tradeoffs is not always valid.
  - EE84M3i 5y ago
    Forgive my ignorance, but could you explain a bit more why frequency scaling needs to be disabled at kernel compile time, and can't be disabled at runtime via sysctl?
    - sudorook 5y ago
      I elided a lot detail for the sake of making a simple example about power consumption. For applications that don't need hard real-time and/or aren't very high frequency, one could set the CPU frequency governor to always run at max frequency. You're right in that this can be done at runtime without a need to disable CONFIG_CPU_FREQ. To achieve hard real-time at high frequency (roughly >10kHz), the standard kernel and PREEMPT_RT are not sufficient. One approach is to use a microkernel that preempts the Linux kernel. RTAI and Xenomai are two frameworks that do this. Processes run through them cannot be preempted by the kernel, so deterministic timings can be much better guaranteed. It's in this case that one would disable CPU scaling (along with Intel pstates, ACPI power management, and other things I don't remember offhand) when compiling the kernel. Changes in CPU frequency can change the frequency of the time counters on the CPU, and that opens up a vector for the vanilla kernel to interfere with the timers used by the microkernel and potentially break real-time. Now, is it true in this latter scenario that one, as I said, "needs to" disable CPU scaling entirely? Now that you've got me thinking about it, probably not. If one guarantees that the 'performance' (i.e. max freq) governor is running at boot, leaving CONFIG_CPU_FREQ enabled may be fine. (The default governor will still need to be changed to the performance one in the kernel config, though.) Setting max frequency at runtime, however, maybe not. This is probably an edge case (I haven't tested this), but if the system boots with a dynamic CPU frequency enabled, the standard kernel, which cannot access information about threads controlled by the microkernel, could scale down the CPU a real-time thread is running on and mess up the timings. Changing the frequency mode at runtime after that happens may not fix that issue. If I were to write that again, I'd use "should" or instead of "needs to." It's much simpler to disable frequency scaling in the menuconfig while disabling everything else than it is to leave it in and worry about it afterward. None of this changes the point about real-time and power consumption, though.
borman 5y ago
> where lowest latency is needed Not so sure it would even be "the lowest", rather than "low enough, stable and predictable", even if it's higher on average than optimal. For example, comparing a hashtable lookup to a BST lookup (and ignoring the memory hierarchy for the sake of example), the former would be faster on average, but the latter would produce more predictable lookup times without hiccups.
duped 5y ago
Slight nit, real time systems are designed for a maximum worst case latency within some spec, not minimum latency. You can make a system that has a lower best case latency but that doesn't matter if the worst case causes a missed deadline. And in the context of real time systems, a missed deadline is either equivalent to a catastrophic error (hard real time) or really terrible but non catastrophic error (soft real time). Soft real time applications like audio (no one dies because of a buffer underrun) can sacrifice the maximum worst case for better average case, because the worst case depends on load and can be worked around. Hard real time cares a lot more about the maximum worst case, like say a CNC machine where missing a deadline can cause a mechanical failure and break the part.
- chippiewill 5y ago
  > a missed deadline is either equivalent to a catastrophic error (hard real time) or really terrible but non catastrophic error (soft real time) I think you're conflating soft/hard real time with criticality (although they're often aligned) Hard real time refers to a total system failure, even if it has trivial consequences. For example if a printer has a timing error at any point the page it's printing is irrecoverably ruined. It's hard real-time but low criticality. Soft real time is where you can miss deadlines but the performance / experience degrades rather than fails. e.g. frame rate drops in a video game. The frames are still valid when delivered late, but the experience is poor. > Slight nit, real time systems are designed for a maximum worst case latency within some spec, not minimum latency. Indeed, in fact at times you don't even want a substantially lower minimum latency because it may cause unreasonable jitter in your output.
chippiewill 5y ago
> I'm not too familiar with kernel development so someone else should give you a definitive answer, but considering the tradeoffs it's very unlikely. I believe most of PREEMPT_RT has already been merged over the years. The expensive bits are all guarded behind a compile time flag.

marcan_42 5y ago

It's already being merged; PREEMPT_RT got merged last year. We're close to the endgame on that front. There are some performance tradeoffs to using an RT kernel; you trade some throughput for determinism. That said, I've been running RT on my desktop most of the time for years and never had any issues.

pabs3 5y ago
After all the patches are merged, will there be work on making PREEMPT_RT a boot-time or run-time choice rather than build-time? Why did you choose RT for your desktop?
- op00to 5y ago
  Id consider RT if I was serious about audio.
- marcan_42 5y ago
  I do audio stuff :)
  - justaj 5y ago
    And luckily _much_ more! (I'm deeply grateful for your contributions btw)
- chippiewill 5y ago
  > will there be work on making PREEMPT_RT a boot-time or run-time choice rather than build-time? Given the performance critical nature of how this changes things I personally think it's unlikely. It does stuff like change the way spinlocks work in the Kernel which would be not only difficult, but expensive without recompilation. The best I think you'll get is a boot-time option to switch between pre-compiled kernels. That being said, most of the RT Linux patches over the years are actually compiled in all the time anyway so a lot of these improvements have been incorporated for a long time.
  - 5y ago