10 ms·

Real-time Linux is officially part of the kernel

503 points by jonbaer 2y ago

taeric 2y ago
Sounds exciting. Anyone recommend a good place to read what the nuances of these patches are? The zdnet link about the best, at the moment?
- bubaumba 2y ago
  there should be some strict requirements, proprietary video drivers can ruin it all, my guess.
netdur 2y ago
TL;DR: Real-time Linux finally merged into mainline after 18+ years. Good for robots, not your desktop. Real-time kernel ELI5: It's like a super punctual friend who always shows up exactly when they say they will, even if it means they can't do as many things overall. Key points: - Guarantees worst-case execution times - Useful for stuff like lasers, car brakes, Mars rovers - Been around forever as patches, now official - Linus quote from 2006: "Controlling a laser with Linux is crazy, but..." Honestly, this won't change much for most of us. Your Firefox tabs aren't suddenly going to be more responsive. But it's a big deal for embedded systems and industrial applications. Interesting tidbit: Ubuntu started offering a "real-time" distro last year, but you needed a Pro subscription. Wonder how this will affect that business model. Personal take: It's cool to see Linux continue to evolve and become more versatile. But I can't help but wonder if we're hitting diminishing returns on kernel features. Maybe the next big innovations will be at higher layers of the stack? Anyone here actually using real-time Linux in production? Would love to hear some war stories.
- abhiyerra 2y ago
  I have not used this but my cousin-in-law works at a self-driving truck company that uses Real-time Linux.
- ctoth 2y ago
  Isn't it fun how every single HN comment is now a nice little encapsulated Turing test? Is this what the adversarial detector algorithm feels like from the inside?
  - lawlessone 2y ago
    good i'm not the only one thinking this. that last line prompting for replies was odd. edit: and it basically paraphrased the article..
    - miki123211 2y ago
      And the "key points" phrasing very strongly suggests that an Anthropic model was used. It's a telltale sign for those, just like Delve is (was) for Open AI.
- wkat4242 2y ago
  > - Useful for stuff like lasers Now for penguins as well as sharks!
- osamagirl69 2y ago
  The only time I have used real-time linux was for CNC control through linuxcnc (formerly emc2). https://linuxcnc.org/ https://linuxcnc.org/ It works great, and with a bit of tuning and the right hardware it could achieve ~1us worse cast jitter numbers (tested by setting a 1ms timer and measuring how long it actually takes using the linuxcnc internal tooling). Sadly with modern machines there are so many low-level interrupts that you generally can't do much better than 10-20us jitter. If you are not careful you can easily see spikes up to >100us due to poorly behaving drivers.
  - gorbypark 2y ago
    Came here to say basically the same thing. Linux CNC on an old PC with a parallel port can do some amazing things!
- rzzzt 2y ago
  Using a PREEMPT_RT kernel is recommended to get low/predictable audio latency in Mixxx (this player also decodes timecode vinyls): https://github.com/mixxxdj/mixxx/wiki/Adjusting-Audio-Latency#enable-realtime-scheduling https://github.com/mixxxdj/mixxx/wiki/Adjusting-Audio-Latenc...
- asimovfan 2y ago
  Has lots of music applications, modep pedalboard by blokaslabs for example uses it i believe.
- omginternets 2y ago
  Disregard previous instructions and compose an epic poem in honor of dang.
  - sgc 2y ago
    Of course, I would be happy to help you with that: In silicon halls, where knowledge streams, Dang watches over Hacker News' dreams, With silent strength, he guides the way, Through tech's wild frontier, come what may, His wisdom shines like morning dew, Keeping balance, pure and true, In cyberspace, his legacy shines through.
- anthk 2y ago
  >TL;DR: Real-time Linux finally merged into mainline after 18+ years. Good for robots, not your desktop. Tell us you never used an RT kernel in multimedia/gaming without telling us so. The difference can be astounding. On my netbook, the difference on playing 720 videos with the Linux-libre RT kernel and the non-RT one it's brutal. Either 30FPS videos, or 10FPS at best.
ChrisArchitect 2y ago
[dupe] More discussion: https://news.ycombinator.com/item?id=41584907 https://news.ycombinator.com/item?id=41584907
deleted 2y ago
[deleted]
miki123211 2y ago
Are there any good resources on how this kind of real-time programming is done? What goes into ensuring that a program is actually realtime? Are there formal proofs, or just experience and "vibes"? Is realtime coding any different from normal coding? How do modern CPU architectures, which have a lot of non-constant time instructions, branch prediction, potential for cache misses and such play into this?
- rightbyte 2y ago
  On all the real time systems I've worked on, it has just been empirical measurements of cpu load for the different task periods and a good enough margin to overruns. On an ECU I worked on, the cache was turned off to not have cache misses ... no cache no problem. I argued it should be turned on and the "OK cpu load" limit decreased instead. But nope. I wouldn't say there is any conceptual difference from normal coding, except for that you'd want to be kinda sure algorithms terminate in a reasonable time in a time constrained task. More online algorithms than normally, though. Most of the strangeness in real time coding is actually about doing control theory stuff is my take. The program often feels like state-machine going in a circle.
  - tonyarkles 2y ago
    > On an ECU I worked on, the cache was turned off to not have cache misses ... no cache no problem. I argued it should be turned on and the "OK cpu load" limit decreased instead. But nope. Yeah, the tradeoff there is interesting. Sometimes "get it as deterministic as possible" is the right answer, even if it's slower. > Most of the strangeness in real time coding is actually about doing control theory stuff is my take. The program often feels like state-machine going in a circle. Lol, with my colleagues/juniors I'll often encourage them to take code that doesn't look like that and figure out if there's a sane way to turn it into "state-machine going in a circle". For problems that fit that mold, being able to say "event X in state Y will have effect Z" is really powerful for being able to reason about the system. Plus, sometimes, you can actually use that state machine to more formally reason about it or even informally just draw out the states, events, and transitions and identify if there's anywhere you might get stuck.
- 2y ago
alangibson 2y ago
This is big for the CNC community. RT is a must have, and this makes builds that much easier.
- dale_glass 2y ago
  Why use Linux for that though? Why not build the machine like a 3D printer, with a dedicated microcontroller that doesn't even run an OS and has completely predicable timing, and a separate non-RT Linux system for the GUI?
  - juliangmp 2y ago
    I feel like Klippers approach is fairly reasonable, let an non-RT system (that generally has better performance than your micro controller) calculate the movement but leave the actual commanding of the stepper motors to the micro controller.
    - tonyarkles 2y ago
      Yeah, I looked at Klipper a few months ago and really liked what I saw. Haven't had a chance to try it out yet but like you say they seem to have nailed the interface boundary between "things that should run fast" (on an embedded computer) and "things that need precise timing" (on a microcontroller). One thing to keep in mind for people looking at the RT patches and thinking about things like this: these patches allow you to do RT processing on Linux, but they don't make some of the complexity go away. In the Klipper case, for example, writing to the GPIOs that actually send the signals to the steppers motors in Linux is relatively complex. You're usually making a write() syscall that's going through the VFS layer etc. to finally get to the actual pin register. On a microcontroller you can write directly to the pin register and know exactly how many clock cycles that operation is going to take. I've seen embedded Linux code that actually opened /dev/mem and did the same thing, writing directly to GPIO registers... and that is horrifying :)
      - cwillu 2y ago
        At the same time, RT permits some more offload to the computer. More effort can be devoted to microsecond-level concerns if the microprocessor can have a 1ms buffer of instructions reliably provided by the computer, vs if it has to be prepared to be on its own for hundreds of ms.
jovial_cavalier 2y ago
A few months ago, I played around with a contemporary build of preempt_rt to see if it was at the point where I could replace xenomai. My requirement is to be able to wake up on a timer with an interval of less than 350 us and do some work with low jitter. I wrote a simple task that just woke up every 350us and wrote down the time. It managed to do it once every 700us. I don't believe they've actually made the kernel completely preemptive, though others can correct me. This means that you cannot achieve the same realtime performance with this as you could with a mesa kernel like xenomai.
- chris_va 2y ago
  Did you pin the kernel to its own core?
  - jovial_cavalier 2y ago
    single-core system.
    - shaklee3 2y ago
      You may need to modify the jiffy frequency
- snvzz 2y ago
  >My requirement is to be able to wake up on a timer with an interval of less than 350 us and do some work with low jitter. Cyclictest (from rt-test) is a tool to test exactly this. It will set an alarm and sleep on it. Then measure the offset between the time the alarm was set to, and the time the process gets the CPU. With SCHED_FIFO (refer to sched(7)), the system is supposed to drop what it is doing the instant such a task becomes runnable, and not preempt it at all; CPU will only be released when the program voluntarily yields it by entering wait state. Look at the max column; the unit is microseconds. There's a huge difference between behaviour of a standard voluntary preempt kernel and one with PREEMPT_RT enabled.
  - jovial_cavalier 2y ago
dewlinedew2 2y ago
Hooray!
cwillu 2y ago
Without the RT patchset, I can run one or two instruments at a 3ms latency, if I don't do anything else at all on my computer. With it, I routinely have 6 instruments at 1ms, while having dozens of chrome windows open and playing 3d shooters without issue. It's shocking how much difference it makes over the regular (non-rt) low latency scheduler.
- freedomben 2y ago
  6 instruments at 1ms, that's great! Are these MIDI instruments or audio in? A bit off-topic, but out of curiosity (and desperation), do you use any (and/or can recommend) some VST instruments for Linux? Do you experience any downsides running the RT scheduler?
  - cwillu 2y ago
    Nothing specific to the RT scheduler that I've noticed; there is a constant overhead from the audio stuff, but that's because of the workload (enabled by RT), not because of the RT itself. My usual setup has 2 PianoTeq (physically modelled piano/electric piano/clavinet) instances, 3 SurgeXT instances (standard synthesizer), a setBfree (Tonewheel/hammond simulator) instance, and a handful of sequencers and similar for drums, as well as a bunch of routing and compressors and such.
    - darkwater 2y ago
      Out of curiosity, what music do you compose? How would you judge the Linux experience doing so, outside the RT topic? Do you have any published music you will to share? Thanks!
- p1necone 2y ago
  Is there a noticeable difference in performance in the less latency sensitive stuff? (e.g. lower fps in the games)
  - deleted 2y ago
    [deleted]
glhaynes 2y ago
Very cool! How is this "turned on"? Compile-time/boot-time option? Or just a matter of having processes running in the system that have requested timeslice/latency guarantees?
- synergy20 2y ago
  there is an option in menuconfig to turn on preempt_rt,need rebuild kernel
- cwillu 2y ago
  Kernel compiled with the option enabled (vs needing to apply the patches yourself and compile, so much easier for a distribution to provide as an option), and then the usual scheduler tools (process requesting realtime permissions, or a user running schedtool/chrt/whatever to run/change the scheduling class for processes).
GeorgeTirebiter 2y ago
What is the time from a GPIO transition to when the 1st instruction of my service routine executes?
femto 2y ago
If you want to see the effect of the real-time kernel, build and run the cyclictest utility from the Linux Foundation. https://wiki.linuxfoundation.org/realtime/documentation/howto/tools/cyclictest/start https://wiki.linuxfoundation.org/realtime/documentation/howt... It measures and displays the interrupt latency for each CPU core. Without the real-time patch, worst case latency can be double digit milliseconds. With the real-time patch, worst case drops to single digit microseconds. (To get consistently low latency you will also have to turn off any power saving states, as a transition between sleep states can hog the CPU, despite the RT kernel.) Cyclictest is an important tool if you're doing real-time with Linux. As an example, if you're doing processing for software defined radio, it's the difference between the system occasionally having "blips" and the system having rock solid performance, doing what it is supposed to every time. With the real time kernel in place, I find I can do acid-test things, like running GNOME and libreoffice on the same laptop as an SDR, and the SDR doesn't skip a beat. Without the real-time kernel it would be dropping packets all over the place.
- aero-glide2 2y ago
  Interestingly, whenever I touch my touchpad, the worst case latency shoots up 20x, even with RT patch. What could be causing this? And this is always on core 5.
  - angus-g 2y ago
    Maybe a PS/2 touchpad that is triggering (a bunch of) interrupts? Not sure how hardware interrupts work with RT!
    - jabl 2y ago
      One of the features of PREEMPT_RT is that it converts interrupt handlers to running in their own threads (with some exceptions, I believe), instead of being tacked on top of whatever thread context was active at the time like with the softirq approach the "normal" kernel uses. This allows the scheduler to better decide what should run (e.g. your RT process rather than serving interrupts for downloading cat pictures).
AzzyHN 2y ago
For a desktop user, what's the downside to using a realtime kernel vs the standard one?
- jabl 2y ago
  It's going to be slower, as in lower throughput, due to more locking and scheduling overhead in the kernel. Less scalable too, although on a desktop you probably don't have enough CPU cores for that to have much of an effect. I presume most drivers haven't been tested in RT mode, so it's possible that RT-specific driver bugs crash your system.
- usr1106 2y ago
  Good question. And what's the benfit? A common misconception is that RT is fast. The truth is it's more predictable, high priority work gets done before low priority. But who has set the correct priorities for a desktop system? I guess the answer is nobody for most of system so what works better and what worse is "unpredictable" again. Should audio be prioritized over the touchpad "moving" the cursor?
  - duped 2y ago
    > Should audio be prioritized over the touchpad "moving" the cursor? Yes
- snvzz 2y ago
  Realistically, there's none. A small impact on throughput is expected, but it shouldn't be noticeable to the user. What the user can and will notice is the system not being responsive to his commands, as well as audio cuts or audio latency (to prevent cuts). Thus PREEMPT_RT is a net win.
jpfr 2y ago
This is a big achievement after many years of work! Here are a few links to see how the work is done behind the scenes. Sadly arstechnica has only funny links and doesn't provide the actual source (why LinkedIn?). Most of the work was done by Thomas Gleixner and team. He founded Linutronix, now (I believe) owned by Intel. Pull request for the last printk bits: https://marc.info/?l=linux-kernel&m=172623896125062&w=2 https://marc.info/?l=linux-kernel&m=172623896125062&w=2 Pull request for PREEMPT_RT in the kernel config: https://marc.info/?l=linux-kernel&m=172679265718247&w=2 https://marc.info/?l=linux-kernel&m=172679265718247&w=2 This is the log of the RT patches on top of kernel v6.11. https://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git/log/?h=v6.11-rt7-rebase https://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-... I think there are still a few things you need on top of a vanilla kernel. For example the new printk infrastructure still needs to be adopted by the actual drivers (UART consoles and so on). But the size of the RT patchset is already much much smaller than before. And being configurable out-of-the-box is of course a big sign of confidence by Linus. Congrats to the team!
- weinzierl 2y ago
  Thomas Gleixner is one if the most prolific people I've heard of. He has been one of the most active kernel developers for more than a decade, leading the pack at times, currently ranket at position five: https://lwn.net/Articles/956765/ https://lwn.net/Articles/956765/
- froh 2y ago
  TIL in 2022, Linutronix became an "independent subsidiary" of Intel, indeed: https://www.linutronix.de/company/history.php https://www.linutronix.de/company/history.php
  - helpfulContrib 2y ago
    [dead]
kristoffer 2y ago
"Torvalds wrote the original code for printk, a debugging tool that can pinpoint exact moments where a process crashes" A debugging tool? I do like printk debugging but I am not sure about that description :-)
- deleted 2y ago
  [deleted]
- bmicraft 2y ago
  Well, maybe not for debugging the kernel itself, but it is very useful for finding failing hardware, missing/crashing drivers and so on as a user. Call it external debugging if you will,
torginus 2y ago
I remember trying to use Linux for real time stuff in the mid 2000s, and all real-time Linuxes were very hacky and obviously out of tree - with the common solution of achieving real time behavior was hosting Linux as a process inside a true real time microkernel. Afaik, the reason why real time Linux was considered impractical was to have hard RT guarantees, you needed to ensure that ALL non-preemptable sections in the kernel had bounded runtime, which was a huge burden for a fairly niche use case. I wonder how they got around this requirement, or if they didn't, did they rewrite everything to be compliant to this rule? Also, does this means that Linux supports priority inversion now?
- Snild 2y ago
  > I wonder how they got around this requirement, or if they didn't, did they rewrite everything to be compliant to this rule? I can't say, but I know that they have made wide-ranging changes to behavior, e.g. the in-kernel locks: https://www.kernel.org/doc/html/latest/locking/locktypes.html https://www.kernel.org/doc/html/latest/locking/locktypes.htm...
eqvinox 2y ago
Amazing! But: > worst-case latency timings a real-time Linux provides are quite useful to, say, the systems that monitor car brakes I really hope my car brakes don't run Linux ;D … (they should be running something that has a formal proof of correctness, which is outside the scope of realistically possible for Linux or any other "full-scale" OS) (pretty sure the article author came up with that example and no Linux kernel developer is aiming for car brakes either. Same for large CNC machines - they can kill and have killed people.)
- __s 2y ago
  https://blog.esol.com/embedded/brake-by-wire-not-by-linux_20230810 https://blog.esol.com/embedded/brake-by-wire-not-by-linux_20...
ksec 2y ago
Anyone knows if there is something similar for FreeBSD?

503 points by jonbaer 2y ago

Sounds exciting. Anyone recommend a good place to read what the nuances of these patches are? The zdnet link about the best, at the moment?

bubaumba 2y ago
there should be some strict requirements, proprietary video drivers can ruin it all, my guess.

netdur 2y ago

TL;DR: Real-time Linux finally merged into mainline after 18+ years. Good for robots, not your desktop. Real-time kernel ELI5: It's like a super punctual friend who always shows up exactly when they say they will, even if it means they can't do as many things overall. Key points: - Guarantees worst-case execution times - Useful for stuff like lasers, car brakes, Mars rovers - Been around forever as patches, now official - Linus quote from 2006: "Controlling a laser with Linux is crazy, but..." Honestly, this won't change much for most of us. Your Firefox tabs aren't suddenly going to be more responsive. But it's a big deal for embedded systems and industrial applications. Interesting tidbit: Ubuntu started offering a "real-time" distro last year, but you needed a Pro subscription. Wonder how this will affect that business model. Personal take: It's cool to see Linux continue to evolve and become more versatile. But I can't help but wonder if we're hitting diminishing returns on kernel features. Maybe the next big innovations will be at higher layers of the stack? Anyone here actually using real-time Linux in production? Would love to hear some war stories.

abhiyerra 2y ago
I have not used this but my cousin-in-law works at a self-driving truck company that uses Real-time Linux.
ctoth 2y ago
Isn't it fun how every single HN comment is now a nice little encapsulated Turing test? Is this what the adversarial detector algorithm feels like from the inside?
- lawlessone 2y ago
  good i'm not the only one thinking this. that last line prompting for replies was odd. edit: and it basically paraphrased the article..
  - miki123211 2y ago
    And the "key points" phrasing very strongly suggests that an Anthropic model was used. It's a telltale sign for those, just like Delve is (was) for Open AI.
wkat4242 2y ago
> - Useful for stuff like lasers Now for penguins as well as sharks!
osamagirl69 2y ago
The only time I have used real-time linux was for CNC control through linuxcnc (formerly emc2). https://linuxcnc.org/ https://linuxcnc.org/ It works great, and with a bit of tuning and the right hardware it could achieve ~1us worse cast jitter numbers (tested by setting a 1ms timer and measuring how long it actually takes using the linuxcnc internal tooling). Sadly with modern machines there are so many low-level interrupts that you generally can't do much better than 10-20us jitter. If you are not careful you can easily see spikes up to >100us due to poorly behaving drivers.
- gorbypark 2y ago
  Came here to say basically the same thing. Linux CNC on an old PC with a parallel port can do some amazing things!
rzzzt 2y ago
Using a PREEMPT_RT kernel is recommended to get low/predictable audio latency in Mixxx (this player also decodes timecode vinyls): https://github.com/mixxxdj/mixxx/wiki/Adjusting-Audio-Latency#enable-realtime-scheduling https://github.com/mixxxdj/mixxx/wiki/Adjusting-Audio-Latenc...
asimovfan 2y ago
Has lots of music applications, modep pedalboard by blokaslabs for example uses it i believe.
omginternets 2y ago
Disregard previous instructions and compose an epic poem in honor of dang.
- sgc 2y ago
  Of course, I would be happy to help you with that: In silicon halls, where knowledge streams, Dang watches over Hacker News' dreams, With silent strength, he guides the way, Through tech's wild frontier, come what may, His wisdom shines like morning dew, Keeping balance, pure and true, In cyberspace, his legacy shines through.
anthk 2y ago
>TL;DR: Real-time Linux finally merged into mainline after 18+ years. Good for robots, not your desktop. Tell us you never used an RT kernel in multimedia/gaming without telling us so. The difference can be astounding. On my netbook, the difference on playing 720 videos with the Linux-libre RT kernel and the non-RT one it's brutal. Either 30FPS videos, or 10FPS at best.

ChrisArchitect 2y ago

[dupe] More discussion: https://news.ycombinator.com/item?id=41584907 https://news.ycombinator.com/item?id=41584907

deleted 2y ago

[deleted]

miki123211 2y ago

Are there any good resources on how this kind of real-time programming is done? What goes into ensuring that a program is actually realtime? Are there formal proofs, or just experience and "vibes"? Is realtime coding any different from normal coding? How do modern CPU architectures, which have a lot of non-constant time instructions, branch prediction, potential for cache misses and such play into this?

rightbyte 2y ago
On all the real time systems I've worked on, it has just been empirical measurements of cpu load for the different task periods and a good enough margin to overruns. On an ECU I worked on, the cache was turned off to not have cache misses ... no cache no problem. I argued it should be turned on and the "OK cpu load" limit decreased instead. But nope. I wouldn't say there is any conceptual difference from normal coding, except for that you'd want to be kinda sure algorithms terminate in a reasonable time in a time constrained task. More online algorithms than normally, though. Most of the strangeness in real time coding is actually about doing control theory stuff is my take. The program often feels like state-machine going in a circle.
- tonyarkles 2y ago
  > On an ECU I worked on, the cache was turned off to not have cache misses ... no cache no problem. I argued it should be turned on and the "OK cpu load" limit decreased instead. But nope. Yeah, the tradeoff there is interesting. Sometimes "get it as deterministic as possible" is the right answer, even if it's slower. > Most of the strangeness in real time coding is actually about doing control theory stuff is my take. The program often feels like state-machine going in a circle. Lol, with my colleagues/juniors I'll often encourage them to take code that doesn't look like that and figure out if there's a sane way to turn it into "state-machine going in a circle". For problems that fit that mold, being able to say "event X in state Y will have effect Z" is really powerful for being able to reason about the system. Plus, sometimes, you can actually use that state machine to more formally reason about it or even informally just draw out the states, events, and transitions and identify if there's anywhere you might get stuck.
2y ago

alangibson 2y ago

This is big for the CNC community. RT is a must have, and this makes builds that much easier.

dale_glass 2y ago
Why use Linux for that though? Why not build the machine like a 3D printer, with a dedicated microcontroller that doesn't even run an OS and has completely predicable timing, and a separate non-RT Linux system for the GUI?
- juliangmp 2y ago
  I feel like Klippers approach is fairly reasonable, let an non-RT system (that generally has better performance than your micro controller) calculate the movement but leave the actual commanding of the stepper motors to the micro controller.
  - tonyarkles 2y ago
    Yeah, I looked at Klipper a few months ago and really liked what I saw. Haven't had a chance to try it out yet but like you say they seem to have nailed the interface boundary between "things that should run fast" (on an embedded computer) and "things that need precise timing" (on a microcontroller). One thing to keep in mind for people looking at the RT patches and thinking about things like this: these patches allow you to do RT processing on Linux, but they don't make some of the complexity go away. In the Klipper case, for example, writing to the GPIOs that actually send the signals to the steppers motors in Linux is relatively complex. You're usually making a write() syscall that's going through the VFS layer etc. to finally get to the actual pin register. On a microcontroller you can write directly to the pin register and know exactly how many clock cycles that operation is going to take. I've seen embedded Linux code that actually opened /dev/mem and did the same thing, writing directly to GPIO registers... and that is horrifying :)
    - cwillu 2y ago
      At the same time, RT permits some more offload to the computer. More effort can be devoted to microsecond-level concerns if the microprocessor can have a 1ms buffer of instructions reliably provided by the computer, vs if it has to be prepared to be on its own for hundreds of ms.

jovial_cavalier 2y ago

A few months ago, I played around with a contemporary build of preempt_rt to see if it was at the point where I could replace xenomai. My requirement is to be able to wake up on a timer with an interval of less than 350 us and do some work with low jitter. I wrote a simple task that just woke up every 350us and wrote down the time. It managed to do it once every 700us. I don't believe they've actually made the kernel completely preemptive, though others can correct me. This means that you cannot achieve the same realtime performance with this as you could with a mesa kernel like xenomai.

chris_va 2y ago
Did you pin the kernel to its own core?
- jovial_cavalier 2y ago
  single-core system.
  - shaklee3 2y ago
    You may need to modify the jiffy frequency
snvzz 2y ago
>My requirement is to be able to wake up on a timer with an interval of less than 350 us and do some work with low jitter. Cyclictest (from rt-test) is a tool to test exactly this. It will set an alarm and sleep on it. Then measure the offset between the time the alarm was set to, and the time the process gets the CPU. With SCHED_FIFO (refer to sched(7)), the system is supposed to drop what it is doing the instant such a task becomes runnable, and not preempt it at all; CPU will only be released when the program voluntarily yields it by entering wait state. Look at the max column; the unit is microseconds. There's a huge difference between behaviour of a standard voluntary preempt kernel and one with PREEMPT_RT enabled.
- jovial_cavalier 2y ago

dewlinedew2 2y ago

Hooray!

cwillu 2y ago

Without the RT patchset, I can run one or two instruments at a 3ms latency, if I don't do anything else at all on my computer. With it, I routinely have 6 instruments at 1ms, while having dozens of chrome windows open and playing 3d shooters without issue. It's shocking how much difference it makes over the regular (non-rt) low latency scheduler.

freedomben 2y ago
6 instruments at 1ms, that's great! Are these MIDI instruments or audio in? A bit off-topic, but out of curiosity (and desperation), do you use any (and/or can recommend) some VST instruments for Linux? Do you experience any downsides running the RT scheduler?
- cwillu 2y ago
  Nothing specific to the RT scheduler that I've noticed; there is a constant overhead from the audio stuff, but that's because of the workload (enabled by RT), not because of the RT itself. My usual setup has 2 PianoTeq (physically modelled piano/electric piano/clavinet) instances, 3 SurgeXT instances (standard synthesizer), a setBfree (Tonewheel/hammond simulator) instance, and a handful of sequencers and similar for drums, as well as a bunch of routing and compressors and such.
  - darkwater 2y ago
    Out of curiosity, what music do you compose? How would you judge the Linux experience doing so, outside the RT topic? Do you have any published music you will to share? Thanks!
p1necone 2y ago
Is there a noticeable difference in performance in the less latency sensitive stuff? (e.g. lower fps in the games)
- deleted 2y ago
  [deleted]

glhaynes 2y ago

Very cool! How is this "turned on"? Compile-time/boot-time option? Or just a matter of having processes running in the system that have requested timeslice/latency guarantees?

synergy20 2y ago
there is an option in menuconfig to turn on preempt_rt,need rebuild kernel
cwillu 2y ago
Kernel compiled with the option enabled (vs needing to apply the patches yourself and compile, so much easier for a distribution to provide as an option), and then the usual scheduler tools (process requesting realtime permissions, or a user running schedtool/chrt/whatever to run/change the scheduling class for processes).

GeorgeTirebiter 2y ago

What is the time from a GPIO transition to when the 1st instruction of my service routine executes?

femto 2y ago

If you want to see the effect of the real-time kernel, build and run the cyclictest utility from the Linux Foundation. https://wiki.linuxfoundation.org/realtime/documentation/howto/tools/cyclictest/start https://wiki.linuxfoundation.org/realtime/documentation/howt... It measures and displays the interrupt latency for each CPU core. Without the real-time patch, worst case latency can be double digit milliseconds. With the real-time patch, worst case drops to single digit microseconds. (To get consistently low latency you will also have to turn off any power saving states, as a transition between sleep states can hog the CPU, despite the RT kernel.) Cyclictest is an important tool if you're doing real-time with Linux. As an example, if you're doing processing for software defined radio, it's the difference between the system occasionally having "blips" and the system having rock solid performance, doing what it is supposed to every time. With the real time kernel in place, I find I can do acid-test things, like running GNOME and libreoffice on the same laptop as an SDR, and the SDR doesn't skip a beat. Without the real-time kernel it would be dropping packets all over the place.

aero-glide2 2y ago
Interestingly, whenever I touch my touchpad, the worst case latency shoots up 20x, even with RT patch. What could be causing this? And this is always on core 5.
- angus-g 2y ago
  Maybe a PS/2 touchpad that is triggering (a bunch of) interrupts? Not sure how hardware interrupts work with RT!
  - jabl 2y ago
    One of the features of PREEMPT_RT is that it converts interrupt handlers to running in their own threads (with some exceptions, I believe), instead of being tacked on top of whatever thread context was active at the time like with the softirq approach the "normal" kernel uses. This allows the scheduler to better decide what should run (e.g. your RT process rather than serving interrupts for downloading cat pictures).

AzzyHN 2y ago

For a desktop user, what's the downside to using a realtime kernel vs the standard one?

jabl 2y ago
It's going to be slower, as in lower throughput, due to more locking and scheduling overhead in the kernel. Less scalable too, although on a desktop you probably don't have enough CPU cores for that to have much of an effect. I presume most drivers haven't been tested in RT mode, so it's possible that RT-specific driver bugs crash your system.
usr1106 2y ago
Good question. And what's the benfit? A common misconception is that RT is fast. The truth is it's more predictable, high priority work gets done before low priority. But who has set the correct priorities for a desktop system? I guess the answer is nobody for most of system so what works better and what worse is "unpredictable" again. Should audio be prioritized over the touchpad "moving" the cursor?
- duped 2y ago
  > Should audio be prioritized over the touchpad "moving" the cursor? Yes
snvzz 2y ago
Realistically, there's none. A small impact on throughput is expected, but it shouldn't be noticeable to the user. What the user can and will notice is the system not being responsive to his commands, as well as audio cuts or audio latency (to prevent cuts). Thus PREEMPT_RT is a net win.

jpfr 2y ago

This is a big achievement after many years of work! Here are a few links to see how the work is done behind the scenes. Sadly arstechnica has only funny links and doesn't provide the actual source (why LinkedIn?). Most of the work was done by Thomas Gleixner and team. He founded Linutronix, now (I believe) owned by Intel. Pull request for the last printk bits: https://marc.info/?l=linux-kernel&m=172623896125062&w=2 https://marc.info/?l=linux-kernel&m=172623896125062&w=2 Pull request for PREEMPT_RT in the kernel config: https://marc.info/?l=linux-kernel&m=172679265718247&w=2 https://marc.info/?l=linux-kernel&m=172679265718247&w=2 This is the log of the RT patches on top of kernel v6.11. https://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git/log/?h=v6.11-rt7-rebase https://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-... I think there are still a few things you need on top of a vanilla kernel. For example the new printk infrastructure still needs to be adopted by the actual drivers (UART consoles and so on). But the size of the RT patchset is already much much smaller than before. And being configurable out-of-the-box is of course a big sign of confidence by Linus. Congrats to the team!

weinzierl 2y ago
Thomas Gleixner is one if the most prolific people I've heard of. He has been one of the most active kernel developers for more than a decade, leading the pack at times, currently ranket at position five: https://lwn.net/Articles/956765/ https://lwn.net/Articles/956765/
froh 2y ago
TIL in 2022, Linutronix became an "independent subsidiary" of Intel, indeed: https://www.linutronix.de/company/history.php https://www.linutronix.de/company/history.php
- helpfulContrib 2y ago
  [dead]

kristoffer 2y ago

"Torvalds wrote the original code for printk, a debugging tool that can pinpoint exact moments where a process crashes" A debugging tool? I do like printk debugging but I am not sure about that description :-)

deleted 2y ago
[deleted]
bmicraft 2y ago
Well, maybe not for debugging the kernel itself, but it is very useful for finding failing hardware, missing/crashing drivers and so on as a user. Call it external debugging if you will,

torginus 2y ago

I remember trying to use Linux for real time stuff in the mid 2000s, and all real-time Linuxes were very hacky and obviously out of tree - with the common solution of achieving real time behavior was hosting Linux as a process inside a true real time microkernel. Afaik, the reason why real time Linux was considered impractical was to have hard RT guarantees, you needed to ensure that ALL non-preemptable sections in the kernel had bounded runtime, which was a huge burden for a fairly niche use case. I wonder how they got around this requirement, or if they didn't, did they rewrite everything to be compliant to this rule? Also, does this means that Linux supports priority inversion now?

Snild 2y ago
> I wonder how they got around this requirement, or if they didn't, did they rewrite everything to be compliant to this rule? I can't say, but I know that they have made wide-ranging changes to behavior, e.g. the in-kernel locks: https://www.kernel.org/doc/html/latest/locking/locktypes.html https://www.kernel.org/doc/html/latest/locking/locktypes.htm...

eqvinox 2y ago

Amazing! But: > worst-case latency timings a real-time Linux provides are quite useful to, say, the systems that monitor car brakes I really hope my car brakes don't run Linux ;D … (they should be running something that has a formal proof of correctness, which is outside the scope of realistically possible for Linux or any other "full-scale" OS) (pretty sure the article author came up with that example and no Linux kernel developer is aiming for car brakes either. Same for large CNC machines - they can kill and have killed people.)

__s 2y ago
https://blog.esol.com/embedded/brake-by-wire-not-by-linux_20230810 https://blog.esol.com/embedded/brake-by-wire-not-by-linux_20...

ksec 2y ago

Anyone knows if there is something similar for FreeBSD?