17 ms·
The real realtime preemption end game
- andy_ppp 3y agoWhat do other realtime OS kernels do when printing from various places? It almost seems like this should be done in hardware because it's such a difficult problem to not lose messages but also have them on a different OS thread in most cases.
- EdSchouten 3y agoAnother option is simply to print less, but expose more events in the form of counters. Unfortunately, within a kernel that’s as big as Linux, that would leave you with many, many, many counters. All of which need to be exported and monitored somehow.
- taeric 3y agoThis seems to imply you would have more counters than messages? Why would that be? That is, I would expect moving to counters to be less information, period. That not the case?
- nraynaud 3y agoMy guess is that each counter would need to have a discovery point, a regular update mechanism and a documentation, while you can send obscure messages willy-nilly in the log? And also they become an Application Interface with a life cycle while (hopefully) not too many people will go parse the log as an API.
- taeric 3y agoI think that makes sense, though I would still expect counters to be more dense than logs. I'm definitely interested in any case studies on this.
- ajross 3y agoIt's just hard, and there's no single answer. In Zephyr, we have a synchronous printk() too, as for low-level debugging and platform bringup that's usually desirable (i.e. I'd like to see the dump from just before the panic please!). For production logging use, though, there is a fancier log system[1] designed around latency boundaries that essentially logs a minimally processed stream to a buffer than then gets flushed from a low priority thread. And this works, and avoids the kinds of problems detailed in the linked article. But it's fiddly to configure, expensive in an RTOS environment (you need RAM for that thread stack and the buffer), depends on having a I/O backend that is itself async/low-latency, and has the mentioned misfeature where when things blow up, it's usually failed to flush the information you need out of its buffer. [1] Somewhat but not completely orthogonal with printk. Both can be implemented in terms of each others, mostly. Sometimes.
- vlovich123 3y agoWhat if the lower priority thread is starved and the buffer is full? Do you start dropping messages? Or overwrite the oldest ones and skip messages?
- ajross 3y agoIt drops messages. That's almost always the desired behavior: you never want your logging system to be doing work when the system is productively tasked with other things. I know there was some level of argument about whether it's best to overwrite older content (ring-buffer-style, probably keeps the most important stuff) or drop messages at input time (faster, probably fewer messages dropped overall). But logging isn't my area of expertise and I forget the details. But again, the general point being that this is a complicated problem with tradeoffs, where most developers up the stack tend to think of it as a fixed facility that shouldn't ever fail or require developer bandwidth. And it's not, it's hard.
- Kenji 3y ago[dead]
- xenadu02 3y agoIn many problem spaces you can optimize for the common success and failure paths if you accept certain losses on long-tail failure scenarios. A common logging strategy is to use a ring buffer with a separate isolated process reading from the ring. The vast majority of the time the ring buffer handles temporary disruptions (eg slow disk I/O to write messages to disk) but in the rare failure scenarios you simply overwrite events in the buffer and increment an atomic overwritten event counter. Events do not get silently dropped but you prioritize forward progress at the cost of data loss in rare scenarios. Microkernels and pushing everything to userspace just moves the tradeoffs around. If your driver is in userspace and blocks writing a log message because the log daemon is blocked or the I/O device it is writing the log to is overloaded it does the same thing. Your realtime thread won't get what it needs from the driver within your time limit. It all comes down to CAP theorem stuff. If you always want the kernel (or any other software) to be able to make forward progress within specific time limits then you must be willing to tolerate some data loss in failure scenarios. How much and how often it happens depends on specific design factors, memory usage, etc.
- TeeMassive 3y agoIt's kind of crazy that a feature necessitated 20 years of active development to be somewhat called complete. I hope it will be ready soon. I'm working in a project that has strict serial communication requirements and it has caused us a lot of headaches.
- eisbaw 3y agoZephyr RTOS.
- worthless-trash 3y agoCan you expand on this, as I'm a little naive in this area, say you isolated the cpus (isolcpus parameter) and then taskset your task onto the isolated cpu, would not the scheduler no longer be involved, and your task be the only thing serviced by that CPU ? Is it other interrupts on the CPU that break your process out of the "real time" requirement, I find this all so interesting.
- TeeMassive 3y agoIt's an embedded system with two logical cores with at least 4 other critical processes running. Doing that will only displace the problem.
- worthless-trash 3y agoI (incorrectly) assumed that serial port control was the highly sensitive time problem that was being dealt with here.
- loeg 3y agoSynchronous logging strikes again! We ran into this some at work with GLOG (Google's logging library), which can, e.g., block on disk IO if stdout is a file or whatever. GLOG was like, 90-99% of culprits when our service stalled for over 100ms.
- cduzz 3y agoI have discussions with cow-orkers around logging; "We have Best-Effort and Guaranteed-Delivery APIs" "I want Guaranteed Delivery!!!" "If the GD logging interface is offline or slow, you'll take downtime; is that okay?" "NO NO Must not take downtime!" "If you need it logged, and can't log it, what do you do?" These days I just point to the CAP theorem and suggest that logging is the same as any other distributed system. Because there's a wikipedia article with a triangle and the word "theorem" people seem to accept that. [edit: added "GD" to clarify that I was referring to the guaranteed delivery logging api, not the best effort logging API]
- loeg 3y agoI read GD as “god damn,” which also seems to fit.
- rezonant 3y agoaw you beat me to it
- msm_ 3y agoInteresting, I'd think logging is one of the clearest situations when you want best effort. Logging is, almost by definition, not the "core" of your application, so failure to log properly should not prevent the core of the program from working. Killing the whole program because logging server is clearly throwing the baby out with the bathwater. What people probably mean is "logging is important, let's avoid losing log messages if possible", which is what "best" in "best effort" stands for. For example it's often a good idea to have a local log queue, to avoid data loss in case of a temporary log server downtime.
- tyingq 3y agoI wonder if this being fixed will result in it displacing some notable amount of made-for-realtime hardware/software combos. Especially since there's now lots of cheap, relatively low power, and high clock rate ARM and x86 chips to choose from. With the clock rates so high, perfect real-time becomes less important as you would often have many cycles to spare for misses. I understand it's less elegant, efficient, etc. But sometimes commodity wins over correctness.
- foobarian 3y agoEthernet nods in vigorous agreement
- tuetuopay 3y agoThe thing is, stuff that require hard realtime cannot satisfy with "many cycles to spare for misses". And CPU cycles is not the whole story. A badly made task could lock down the kernel not doing anything useful. The point of hard realtime is "nothing cannot prevent this critical task from running". For automotive and aerospace, you really want the control systems to be able to run no matter what.
- tyingq 3y agoYes, there are parts of the space that can't be displaced with this. I'm unclear on why you put "many cycles to spare for misses" in quotes, as if it's unimportant. If a linux/arm (or x86) solution is displacing a much lower speed "real real time" solution, that's the situation...the extra cycles mean you can tolerate some misses while still being as granular as what you're replacing. Not for every use case, but for many.
- bee_rider 3y agoIt is sort of funny that language has changed to the point where quotes are assumed to be dismissive or sarcastic. Maybe they used the quotes because they were quoting you, haha.
- 3y ago
- eisbaw 3y agoGreat to hear. However even if Linux the kernel is real-time, likely the hardware won't be due to caches and internal magic CPU trickery. Big complex hardware is a no-no for true real-time. That's why AbsInt and WCET tools mainly has simple CPU architectures. 8051 will truly live forever. btw, Zephyr RTOS.
- wholesomepotato 3y agoFeatures of modern CPUs don't really prevent them from real time usage, afaik. As long as something is bounded and can be reasoned about it can be used to build a real time system. You can always assume no cache hits and alikes, maximum load etc and as long as you can put a bound on the time it will take, you're good to go.
- synergy20 3y agomlock your memory, test with cache miss and cache invalidation scenarios will help, using no heap for memory allocation, but it's a bit hard
- jeffreygoesto 3y agoRight. But still possible. https://www.etas.com/en/applications/etas-middleware-solution.php https://www.etas.com/en/applications/etas-middleware-solutio...
- eschneider 3y agoDoes anyone use paged memory in hard realtime systems?
- SAI_Peregrinus 3y agoExactly. "Real-time" is a misnomer, it should be called "bounded-time". As long as the bound is deterministic, known in advance, and guaranteed, it's "real-time". For it to be useful it also must be under some application-specific duration. The bounds are usually in CPU cycles, so a faster CPU can sometimes be used even if it takes more cycles. CPUs capable of running Linux usually have higher latency (in cycles) than microcontrollers, but as long as that can be kept under the (wall clock) duration limits with bounded-time it's fine. There will still be cases where the worst-case latency to fetch from DRAM in an RT-Linux system will be higher than a slower MCU fetching from internal SRAM, so RT-Linux won't take over all these systems.
- Aaargh20318 3y agoWhat does this mean for the common user? Is this something you would only enable in very specific circumstances or can it also bring a more responsive system to the general public?
- stavros 3y agoAs far as I can understand, this is for Linux becoming an option when you need an RTOS, so for critical things like aviation, medical devices, and other such systems. It doesn't do anything for the common user.
- SubjectToChange 3y agoThe Linux kernel, real-time or not, is simply too large and too complex to realistically certify for anything safety critical.
- ska 3y agoFor the parts of such systems that you would need an RTOS for this isn't really a likely replacement because the OS is way too complex. The sort of thing it could help with is servicing hardware that does run hard realtime. For example, you have an RTOS doing direct control of a robot or medical device or whatever, and you have a UI pendant or the like that a user is interacting with. If linux on that pendant can make some realtime latency guarantees, you may be able to simplify communication between the two without risking dropping bits on the floor. Conversely, for the common user it could improve things like audio/video streaming, in theory but I haven't looked into details or how much trouble there is currently.
- elcritch 3y agoIt depends on the field. I know of one robots control software company planning to switch to a RT Linux stack. Their current one is a *BSD derived rtos that runs, kid you not, alongside windows. RT Linux might not pass on some certifications, but there’s likely many systems where it would be sufficient.
- knorker 3y agoI just want SCHED_IDLEPRIO to actually do what it says.
- deepsquirrelnet 3y agoWhat a blast from the past. I compiled a kernel for Debian with RT_PREEMPT about 17-18 years ago to use with scientific equipment that needed tighter timings. I was very impressed at the latencies and jitter. I haven’t really thought about it since then, but I can imagine lots of used cases for something like an embedded application with raspberry pi where you don’t quite want to make the leap into a microcontroller running an RTOS.
- HankB99 3y agoInteresting to mention the Raspberry Pi. I saw an article just a day or two ago that claimed that the RpiOS was stated by and ran on top of RTOS. That's particularly interesting because at one time years ago, I saw suggestions that Linux could run as a task on an RTOS. Things that required hard real time deadlines could run on the RTOS and not be subject to the delays that a virtual memory system could entail. I don't recall if this was just an idea or was actually implemented. I also have seen only the one mention of RpiOS on an RTOS so I'm curious about that.
- rsaxvc 3y ago>That's particularly interesting because at one time years ago, I saw suggestions that Linux could run as a task on an RTOS. I've worked with systems that ran Linux as a task of uITRON as well as threadX, both on somewhat obscure ARM hardware. Linux managed the MMU but had a large carveout for the RTOS code. They had some strange interrupt management so that Linux could 'disable interrupts' but while Linux IRQs were disabled, an RTOS IRQ could still fire and context switch back to an RTOS task. I haven't seen anything like this on RPi though, but it's totally doable.
- 0xDEF 3y agoWhat do embedded real-time Linux people use for bootloader, init system, utilities, and C standard library implementation? Even Android that does not have real-time constraints ended up using Toybox for utilities and rolling their own C standard library (Bionic).
- rcxdude 3y agoYou aren't likely to need to change a lot of these: the whole point is basically making it so that all that can run as normal but won't really get in the way of your high-priority process. It's just that your high-priority process needs to be careful not to block on anything that might take too long due to some other stuff running. In which case you may need to avoid certain C standard library calls, but not replace it entirely.
- jovial_cavalier 3y agoI use u-boot for a boot loader. As for init and libc, I just use systemd and glibc. Boot time is not a bottleneck for my application (however long it takes, the client will take longer…), and I’m sure there’s some more optimal libc to use, but I’m not sure the juice is worth the squeeze. I’m also interested in what others are doing.
- shandor 3y agoI guess U-boot, uclibc, and busybox is quite common starting point. Of course, this varies immensely between different use cases, as ”embedded Linux” spans such a huge swath of different kinds of systems from very cheap and simple to complex and powerful.
- salamanderman 3y agoI had a frustrating number of job interviews in my early career where the interviewers didn't know what realtime actually was. That "and predictable delay" concept from the article frequently seemed to be lost on many folks, who seemed to think realtime just meant fast, whatever that means.
- mort96 3y agoI would even remove the "minimum" part altogether; the point of realtime is that operations have predictable upper bounds. That might even mean slower average cases than in non-realtime systems. If you're controlling a car's braking system, "the average delay is 50ms but might take up to 80ms" might be acceptable, whereas "the average delay is 1ms but it might take arbitrarily long, possibly multiple seconds" isn't.
- ska 3y agoThe old saying "real time" /= "real fast". Hard vs "soft" realtime muddies things a bit, but I think it's probably the majority of software developers don't really understand what realtime actually is either.
- biorach 3y ago[flagged]
- deleted 3y ago[deleted]
- NalNezumi 3y agoSlightly tangential, but does anyone know good learning material to understand real-time (Linux) kernel more? For someone with rudimentary Linux knowledge. I've had to compile&install real-time kernel as a requirement for a robot arm (franka) control computer. It would be nice to know a bit more than just how to install the kernel.
- ActorNightly 3y agohttps://www.freertos.org/implementation/a00002.html https://www.freertos.org/implementation/a00002.html Generally, having experience with Greenhills in a previous job, for personal projects like robotics or control systems I would recommend programming a microcontroller directly rather than dealing with SoC with RTOS. Modern STM32s with Cortex chips have enough processing power to run pretty much anything.
- alangibson 3y agoVery exiting news for those of us building CNC machines with LinuxCNC. The end of kernel patches is nigh!
- Tomte 3y agoOSADL runs a cool QA farm: https://www.osadl.org/OSADL-QA-Farm-Real-time.linux-real-time.0.html https://www.osadl.org/OSADL-QA-Farm-Real-time.linux-real-tim...
- Animats 3y agoQNX had this right decades ago. The microkernel has upper bounds on everything it does. There are only a few tens of thousands of lines of microkernel code. All the microkernel does is allocate memory, dispatch the CPU, and pass messages between processes. Everything else, including drivers and loggers, is in user space and can be preempted by higher priority threads. The QNX kernel doesn't do anything with strings. No parsing, no formatting, no messages. Linux suffers from being too bloated for real time. Millions of lines of kernel, all of which have to be made preemptable. It's the wrong architecture for real time. So it took two decades to try to fix this.
- matheusmoreira 3y agoAnd yet it's getting done! It's very impressive work.
- vacuity 3y agoFor a modern example, there's seL4. I believe it does no dynamic memory allocation. It's also formally verified for various properties. (Arguably?) its biggest contribution to kernel design is the pervasive usage of capabilities to securely but flexibly export control to userspace.
- adastra22 3y agoCapabilities are important, but I don’t think that was introduced by seL4. Mach (which underlies macOS) has the same capability-based system.
- thrway63245 3y ago[dead]
- the8472 3y agoFor an example how far the kernel goes to get log messages out even on a dying system and how that's used in real deployments: https://netflixtechblog.com/kubernetes-and-kernel-panics-ed620b9c6225#2965 https://netflixtechblog.com/kubernetes-and-kernel-panics-ed6...
- rwmj 3y agoAbout printk, the backported RT implementation of printk added to the RHEL 9.3 kernel has deadlocks ... https://issues.redhat.com/browse/RHEL-15897 https://issues.redhat.com/browse/RHEL-15897 & https://issues.redhat.com/browse/RHEL-9380 https://issues.redhat.com/browse/RHEL-9380
- w10-1 3y agoThere is no end game until there are end users beating on the system. That would put the 'real' in 'real-time'. But who using a RTOS now would take the systems-integration cost/risk of switching? Would this put Android closer to Metal performance?
- sesm 3y agoIMO if you really care about certain process being responsive, you should allocate dedicated CPU cores and a contiguous region of memory to it, that shouldn’t be touched by the rest of OS. Oh, and also give a it direct access to a separate network card. I’m not sure if Linux supports this.
- pardoned_turkey 3y agoThe conversation here focuses on a distinction between "hard" real-time applications, where you probably don't want a general-purpose OS like Linux no matter what; and "soft" real-time applications like videoconferencing or audio playback, where you nothing terrible happens if you get a bit of stuttering or drop a couple of frames every now and then. The argument is that RT Linux would be a killer solution for that. But you can do all these proposed "soft" use cases with embedded Linux today. It's not like low-latency software video or audio playback is not possible, or wasn't possible twenty years ago. You only run into problems on busy systems where non-preemptible I/O could regularly get in the way. That's seldom a concern in embedded environments. I think there are compelling reasons for making the kernel fully-preemptible, giving people more control over scheduling, and so forth. But these reasons have relatively little to do with wanting Linux to supersede minimalistic realtime OSes or bare-metal code. It's just good hygiene that will result in an OS that, even in non-RT applications, behaves better under load.
- jovial_cavalier 3y agodoes HN have any thoughts on Xenomai[1]? I've been using it for years without issue. On a BeagleBone Black, it typically gives jitter on the order of hundreds of nanoseconds. I would consider it "hard" real-time (as do they). I'm able to schedule tasks periodically on the scale of tens of microseconds, and they never get missed. It differs from this in that Real-Time Linux attempts to make Linux itself preemptive, whereas Xenomai is essentially its own kernel, running Linux as a task on top. It provides an ABI which allows you to run your own tasks alongside or at higher prio than Linux. This sidesteps the `printk()` issue, for instance, since Xenomai doesn't care. It will gladly context switch out of printk in order to run your tasks. The downside is that you can't make normal syscalls while inside of the Xenomai context. Well... you can, but obviously this invalidates the realtime model. For example, calling `printf()` or `malloc()` inside of a xenomai task is not preemptable. The Xenomai ABI does its best to replicate everything you may need as far as syscalls, which works great as long as you're happy doing your own heap allocations. [1]: https://xenomai.org/ https://xenomai.org/
- dataflow 3y agoI feel like focusing on the kernel side misses CPU level issues. Is there any known upper bound on, say, how long a memory access instruction takes on x86?
- rsaxvc 3y agoI don't know for x86. But for things that really matter, I've tested by configuring the MMU to disable caching for the memory that the realtime code lives in and uses to emulate 0% hitrate. And there's usually still a fair amount of variance on top of that depending on if the memory controller has a small cache, and where the memory controller is in its refresh cycle.
- dataflow 3y agoYeah. And I'm not sure that even that would give you the worst case as far as the cache is concerned. Of course I don't know how these implementations work, but it seems plausible that code that directly uses memory could run faster than code that encounters a cache miss beforehand (or contention, if you're using multiple cores). Moreover there's also the instruction cache, and I'm not sure if you can disable caching for that in a meaningful way? For soft real time, I don't see a problem. But for hard real time, it seems a bit scary.
- rsaxvc 3y agoYou're right! I can think of two cases I've run into where bypassing the cache can be faster compared to a miss. On some caches the line must be filled before allowing a write(ignoring any write buffer at the interface above the cache) - those basically halve the memory bandwidth when writing to a lot of cache lines. Some systems now have instructions for filling a cache line directly to avoid this. And some CPUs have bit-per-byte validity tracking to avoid this too. Even on caches with hit-during-fill, a direct read from an address near the last-to-be-filled end of a cacheline can sometimes be a little faster than a cache miss, since the miss will fill the rest of the line first.
- rsaxvc 3y ago