6 ms·

The thing is, stuff that require hard realtime cannot satisfy with "many cycles to spare for misses". And CPU cycles is not the whole story. A badly made task c

The thing is, stuff that require hard realtime cannot satisfy with "many cycles to spare for misses". And CPU cycles is not the whole story. A badly made task could lock down the kernel not doing anything useful. The point of hard realtime is "nothing cannot prevent this critical task from running". For automotive and aerospace, you really want the control systems to be able to run no matter what.

tyingq 3y ago
Yes, there are parts of the space that can't be displaced with this. I'm unclear on why you put "many cycles to spare for misses" in quotes, as if it's unimportant. If a linux/arm (or x86) solution is displacing a much lower speed "real real time" solution, that's the situation...the extra cycles mean you can tolerate some misses while still being as granular as what you're replacing. Not for every use case, but for many.
- bee_rider 3y ago
  It is sort of funny that language has changed to the point where quotes are assumed to be dismissive or sarcastic. Maybe they used the quotes because they were quoting you, haha.
  - tuetuopay 3y ago
    it's precisely why I quoted the text, to quote :)
- archgoon 3y ago
  I'm pretty sure they were just putting it in quotes because it was the expression you used, and they thus were referencing it.
- tuetuopay 3y ago
  You won't be saved from two tasks deadlocking with cycles/second. this is what hard realtime systems are about. However, I do agree that not all systems have a real hard realtime requirements. But those usually can handle a non-rt kernel. As for the quotes, it was a direct citation, not a way to dismiss what you said.
  - tremon 3y ago
    I don't think realtime anything has much to do with mutex deadlocks, those are pretty much orthogonal concepts. In fact, I would make a stronger claim: if your "realtime" system can deadlock, it's either not really realtime or it has a design flaw and should be sent back to the drawing board. It's not like you can say "oh, we have a realtime kernel now, so deadlocks are the kernel's problem". Actual realtime systems are about workload scheduling that takes into account processing deadlines. Hard realtime systems can make guarantees about processing latencies, and can preemptively kill or skip tasks if the result would arrive too late. But this is not something that the Linux kernel can provide, because it is a system property rather than about just the kernel: you can't provide any hard guarantees if you have no time bounds for your data processing workload. So any discussion about -rt in the context of the Linux kernel will always be about soft realtime only.
    - tuetuopay 3y ago
      much agreed. I used deadlocks as an extreme example that's easy to reason about and straight to the point of "something independent of cpu cycles". something more realistic would be IO operations taking more time than expected. you would not want this to be blocking execution for hard rt tasks. In the case of the kernel, it is indeed too large to be considered hard realtime. Best case we can make it into a firmer realtime than it currently is. But I would place it nowhere near avionics flight calculators (like fly-by-wire systems).
    - hamilyon2 3y ago
      I had an introductory course on OS and learned about hard real-time systems. I had impression hard real-time is about memory, deadlocks, livelocks, starvation, and so on. And in general about how to design system that moves forward even in presence of serious bugs and unplanned-for circumstances.
      - syntheweave 3y ago
        Bugs related to concurrency - which is where you get race conditions and deadlocks - tend to pop up wherever there's an implied sequence of dependencies to complete the computation, and the sequence is determined dynamically by an algorithm. For example, if I have a video game where there's collision against the walls, I can understand this as potentially colliding against "multiple things simultaneously", since I'm likely to describe the scene as a composite of bounding boxes, polygons, etc. But to get an answer for what to do in response when I contact a wall, I have to come up with an algorithm that tests all the relevant shapes or volumes. The concurrency bug that appears when doing this in a naive way is that I test one, give an answer to that, then modify the answer when testing the others. That can lead to losing information and "popping through" a wall. And the direction in which I pop through depends on which one is tested first. The conventional gamedev solution to that is to define down the solution set so that it no longer matters which order I test the walls in: with axis aligned boxes, I can say "move only the X axis first, then move only the Y axis". Now there is a fixed order, and a built-in bias to favor one or the other axis. But this is enough for the gameplay of your average platforming game. The generalization on that is to describe it as a constraint optimization problem: there are some number of potential solutions, and they can be ranked relative to the "unimpeded movement" heuristic, which is usually desirable when clipping around walls. That solution set is then filtered down through the collision tests, and the top ranked one becomes the answer for that timestep. Problems of this nature come up with resource allocation, scheduling, etc. Some kind of coordinating mechanism is needed, and OS kernels tend to shoulder a lot of the burden for this. It's different from real-time in that real-time is a specification of what kind of performance constraint you are solving for, vs allowing any kind of performance outcome that returns acceptable concurrent answers.
- nine_k 3y ago
  How much more expensive and power-hungry an ARM core would be, if it displaces a lower-specced core? I bet there are hard-realtime (commercial) OSes running on ARM, and the ability to use a lower-specced (cheaper, simpler, consuming less power) core may be seen as an advantage enough to pay for the OS license.
  - lmm 3y ago
    > How much more expensive and power-hungry an ARM core would be, if it displaces a lower-specced core? The power issue is real, but it might well be the same price or cheaper - a standard ARM that gets stamped out by the million can cost less than a "simpler" microcontroller with a smaller production run.
zmgsabst 3y ago
What’s an example of a system that requires hard real time and couldn’t cope with soft real time on a 3GHz system having 1000 cycle misses costing 0.3us?
- lelanthran 3y ago
  > What’s an example of a system that requires hard real time and couldn’t cope with soft real time on a 3GHz system having 1000 cycle misses costing 0.3us? Any system that deadlocks.
- LeifCarrotson 3y ago
  We've successfully used a Delta Tau real-time Linux motion controller to run a 24 kHz laser galvo system. It's ostensibly good for 25 microsecond loop rates, and pretty intolerant of jitter (you could delay a measurement by a full loop period if you're early). And the processor is a fixed frequency Arm industrial deal that only runs at 1.2 GHz. Perhaps even that's not an example of such a system, 0.3 microseconds is close to the allowable real-time budget, and QC would probably not scrap a $20k part if you were off by that much once. But in practice, every time I've heard "soft real time" suggested, the failure mode is not a sub-microsecond miss but a 100 millisecond plus deadlock, where a hardware watchdog would be needed to drop the whole system offline and probably crash the tool (hopefully fusing at the tool instead of destroying spindle bearings, axis ball screws, or motors and gearboxes) and scrap the part.
  - zmgsabst 3y ago
    Thanks for the detailed reply! I’m trying to understand where the roadblock on a rPi + small FPGA hybrid board for $50 fails at the task… and it sounds like the OS/firmware doesn’t suffice. (Or a SoC, like a Zynq.) Eg, if we could guarantee that the 1.5GHz core won’t “be off” by more than 1us on responding and the FPGA can manage IO directly to buffer out (some of) the jitter, then the cost of many hobby systems with “(still not quite) hard” real time systems would come down to reasonable.
    - rmu09 3y ago

tyingq 3y ago

Yes, there are parts of the space that can't be displaced with this. I'm unclear on why you put "many cycles to spare for misses" in quotes, as if it's unimportant. If a linux/arm (or x86) solution is displacing a much lower speed "real real time" solution, that's the situation...the extra cycles mean you can tolerate some misses while still being as granular as what you're replacing. Not for every use case, but for many.

bee_rider 3y ago
It is sort of funny that language has changed to the point where quotes are assumed to be dismissive or sarcastic. Maybe they used the quotes because they were quoting you, haha.
- tuetuopay 3y ago
  it's precisely why I quoted the text, to quote :)
archgoon 3y ago
I'm pretty sure they were just putting it in quotes because it was the expression you used, and they thus were referencing it.
tuetuopay 3y ago
You won't be saved from two tasks deadlocking with cycles/second. this is what hard realtime systems are about. However, I do agree that not all systems have a real hard realtime requirements. But those usually can handle a non-rt kernel. As for the quotes, it was a direct citation, not a way to dismiss what you said.
- tremon 3y ago
  I don't think realtime anything has much to do with mutex deadlocks, those are pretty much orthogonal concepts. In fact, I would make a stronger claim: if your "realtime" system can deadlock, it's either not really realtime or it has a design flaw and should be sent back to the drawing board. It's not like you can say "oh, we have a realtime kernel now, so deadlocks are the kernel's problem". Actual realtime systems are about workload scheduling that takes into account processing deadlines. Hard realtime systems can make guarantees about processing latencies, and can preemptively kill or skip tasks if the result would arrive too late. But this is not something that the Linux kernel can provide, because it is a system property rather than about just the kernel: you can't provide any hard guarantees if you have no time bounds for your data processing workload. So any discussion about -rt in the context of the Linux kernel will always be about soft realtime only.
  - tuetuopay 3y ago
    much agreed. I used deadlocks as an extreme example that's easy to reason about and straight to the point of "something independent of cpu cycles". something more realistic would be IO operations taking more time than expected. you would not want this to be blocking execution for hard rt tasks. In the case of the kernel, it is indeed too large to be considered hard realtime. Best case we can make it into a firmer realtime than it currently is. But I would place it nowhere near avionics flight calculators (like fly-by-wire systems).
  - hamilyon2 3y ago
    I had an introductory course on OS and learned about hard real-time systems. I had impression hard real-time is about memory, deadlocks, livelocks, starvation, and so on. And in general about how to design system that moves forward even in presence of serious bugs and unplanned-for circumstances.
    - syntheweave 3y ago
      Bugs related to concurrency - which is where you get race conditions and deadlocks - tend to pop up wherever there's an implied sequence of dependencies to complete the computation, and the sequence is determined dynamically by an algorithm. For example, if I have a video game where there's collision against the walls, I can understand this as potentially colliding against "multiple things simultaneously", since I'm likely to describe the scene as a composite of bounding boxes, polygons, etc. But to get an answer for what to do in response when I contact a wall, I have to come up with an algorithm that tests all the relevant shapes or volumes. The concurrency bug that appears when doing this in a naive way is that I test one, give an answer to that, then modify the answer when testing the others. That can lead to losing information and "popping through" a wall. And the direction in which I pop through depends on which one is tested first. The conventional gamedev solution to that is to define down the solution set so that it no longer matters which order I test the walls in: with axis aligned boxes, I can say "move only the X axis first, then move only the Y axis". Now there is a fixed order, and a built-in bias to favor one or the other axis. But this is enough for the gameplay of your average platforming game. The generalization on that is to describe it as a constraint optimization problem: there are some number of potential solutions, and they can be ranked relative to the "unimpeded movement" heuristic, which is usually desirable when clipping around walls. That solution set is then filtered down through the collision tests, and the top ranked one becomes the answer for that timestep. Problems of this nature come up with resource allocation, scheduling, etc. Some kind of coordinating mechanism is needed, and OS kernels tend to shoulder a lot of the burden for this. It's different from real-time in that real-time is a specification of what kind of performance constraint you are solving for, vs allowing any kind of performance outcome that returns acceptable concurrent answers.
nine_k 3y ago
How much more expensive and power-hungry an ARM core would be, if it displaces a lower-specced core? I bet there are hard-realtime (commercial) OSes running on ARM, and the ability to use a lower-specced (cheaper, simpler, consuming less power) core may be seen as an advantage enough to pay for the OS license.
- lmm 3y ago
  > How much more expensive and power-hungry an ARM core would be, if it displaces a lower-specced core? The power issue is real, but it might well be the same price or cheaper - a standard ARM that gets stamped out by the million can cost less than a "simpler" microcontroller with a smaller production run.

zmgsabst 3y ago

What’s an example of a system that requires hard real time and couldn’t cope with soft real time on a 3GHz system having 1000 cycle misses costing 0.3us?

lelanthran 3y ago
> What’s an example of a system that requires hard real time and couldn’t cope with soft real time on a 3GHz system having 1000 cycle misses costing 0.3us? Any system that deadlocks.
LeifCarrotson 3y ago
We've successfully used a Delta Tau real-time Linux motion controller to run a 24 kHz laser galvo system. It's ostensibly good for 25 microsecond loop rates, and pretty intolerant of jitter (you could delay a measurement by a full loop period if you're early). And the processor is a fixed frequency Arm industrial deal that only runs at 1.2 GHz. Perhaps even that's not an example of such a system, 0.3 microseconds is close to the allowable real-time budget, and QC would probably not scrap a $20k part if you were off by that much once. But in practice, every time I've heard "soft real time" suggested, the failure mode is not a sub-microsecond miss but a 100 millisecond plus deadlock, where a hardware watchdog would be needed to drop the whole system offline and probably crash the tool (hopefully fusing at the tool instead of destroying spindle bearings, axis ball screws, or motors and gearboxes) and scrap the part.
- zmgsabst 3y ago
  Thanks for the detailed reply! I’m trying to understand where the roadblock on a rPi + small FPGA hybrid board for $50 fails at the task… and it sounds like the OS/firmware doesn’t suffice. (Or a SoC, like a Zynq.) Eg, if we could guarantee that the 1.5GHz core won’t “be off” by more than 1us on responding and the FPGA can manage IO directly to buffer out (some of) the jitter, then the cost of many hobby systems with “(still not quite) hard” real time systems would come down to reasonable.
  - rmu09 3y ago