4 ms·

> The best case latency is higher than doing the work at interrupt level One approach is to do everything in ISRs, a la RTIC. That requires efficient, vectore

by topspin 1mo ago

> The best case latency is higher than doing the work at interrupt level One approach is to do everything in ISRs, a la RTIC. That requires efficient, vectored, nested, tail-chained, base priority-ed interrupt silicon, and a lot of it, but it is feasible and elegant where this exists, such as Cortex NVIC. Emerging RISC-V devices with xCLIC (ch32v, gd32v, newer ESP32 and others) are potentially even better. I really appreciate that the author took the time to add the Embassy vs RTIC addendum.

monocasa 1mo ago
FWIW, just having a mask in the interrupt controller is normally enough to give you the same thing at the cost of a dozen or so cycles in the critical path. Basically you just keep a mask per priority that can be built up cheaply at init time (or even compile time if you're cute about it), you apply the appropriate mask in the interrupt prologues and epilogues, and pretty much as soon as you apply the new mask in the prologue you go ahead and acknowledge the interrupt.
- topspin 1mo ago
  I am aware. That "dozen or so" is a problem: when everything is an interrupt, there are no interrupts: it's just scheduling, and things that must be scheduled frequently can't suffer "a dozen or so" overhead. For the SRP model to really hum, you need the silicon that solves this.
  - monocasa 1mo ago
    I've found that it doesn't matter except for something that you want at the absolute highest priority anyway, which then by definition doesn't need to jump through the same hoops because nothing can preempt it anyway.
    - topspin 27d ago
      It's more about frequency than priority. When something has to be serviced tens of thousands of times a second, "a dozen or so" becomes a problem. If you have the silicon that solves this, you can retain the model. If you don't then you have to resort to workarounds. Fortunately we typically have more efficient means to deal with such hardware, but not always, and it would be a shame to break the intended model when this is the case.
- amluto 1mo ago
  You can do this on x86 as well at a cost a merely tens to hundreds (possibly lots of hundreds) of thousands of cycles. This is part of why x86 is so popular in the embedded space. (I’m being sarcastic, obviously. x86 interrupts and interrupt returns are hilariously slow. FRED may improve this by quite a bit.)
  - nomel 1mo ago
    What's the technical reason for them being slow? Book keeping with caches or something?
    - monocasa 1mo ago
      Mostly tons of speculative state that needs to be unwound, combined with spectre mitigations, plus tons of committed state that the interrupt prologue needs to save, plus a huge song and dance to do that correctly (that FRED should help with). All combined with the fact that there's a good chance the memory the interrupt handler is going to touch isn't in the cached working set anymore, both in the actual L* caches and in subtler places like the branch predictors and TLBs.
      - rkagerer 1mo ago
        Thanks. Surely this was a performance hit even before Spectre? Also, any good technical resources that concisely describe FRED? I found this, which isn't bad but it's a bit more dumbed down than I'd like: https://www.tomshardware.com/pc-components/cpus/amd-adopts-fred-together-with-intel-for-zen-6-architecture-replacement-for-decades-old-idt-can-improve-performance-and-stability https://www.tomshardware.com/pc-components/cpus/amd-adopts-f...
      - amluto 1mo ago
        You’re missing the big ones: both the interrupt delivery and the IRET (interrupt return) mechanisms use incredibly complicated data structures to determine what the new state should be. They need to dig around in the IDT, the GDT, the TSS and possibly the LDT to find all the register values they need to set, and they need to handle all kinds of backwards compatibility. And they “serialize”, which is an extra heavyweight fence, although that only likely accounts for a few hundred cycles in each direction. Check out the pseudocode in the SDM — there are pages of it, and the pseudocode isn’t even complete. FRED simplifies the state transitions such that the new state is mostly a foregone conclusion based on MSR contents.
jacquesm 1mo ago
> One approach is to do everything in ISRs, a la RTIC. That only works for really simple systems. On more complex systems there is a pretty good chance you will end up with locked up hardware if your ISR is long enough. Interrupts need servicing to keep the data flowing, prioritization is a job for the OS, not the hardware.
- monocasa 1mo ago
  The model works pretty well up to much larger systems than you'd expect. If a particular interrupt has a hard real time constraint, it sounds like a great candidate for a higher priority interrupt which will let it meet that requirement. The biggest constraint is that this is really a single core model. You need something different if you go to SMP. Though there, AMP where the main core runs this 'interrrupt controller is your scheduler' scheme, and the other cores run against a work stealing scheduler for compute bound work items still is a very nice system to program against.
- crest 1mo ago
  The ARM v7M architecture helps a lot here because the NVIC supports priority inheritance without disabling interrupts. Yes you have to assign your priorities correctly and yield CPU time for anything compute bound, but again the architecture helps you in the form of the SVCall+PendSV exception pair. The PendSV handler can easily be multiplexed on the Cortex-M3/M4 using atomic operations on bit-banded memory. I've needed only a few dozen lines of Forth to set this up on bare metal without an RTOS to split peripheral drivers in a short interrupt handler saving the required state and delegating protocol handling to a lower priority callback.
- topspin 1mo ago
  > On more complex systems there is a pretty good chance you will end up with locked up hardware if your ISR is long enough. That is true of all such systems, from MCUs to the greatest CPUs ever made, and all prevailing software stacks running on them. Nothing about RTIC precludes servicing interrupts in a timely manner. It is based on a mathematical model of concurrency called Stack Resource Policy (SRP) and is entirely capable of reliably implementing whatever interrupt regime you imagine your "complex" system requires, with zero risk of priority inversion. > prioritization is a job for the OS, not the hardware So say you. The long and storied history of fragility and failure related to scheduling and interrupt handling suggests there is opportunity for greater rigor than the conventional muddle you assert as best.
mrheosuper 1mo ago
How would it handle kernel/user space if everything runs inside ISR context ?
- topspin 1mo ago
  Whatever OS/applications you have run in low priority ISR contexts, taking however much time they wish. When a high priority interrupt happens, the lower priority ISR contexts are preempted and then resumed after the high priority ISR is complete. Go have a look at the design and the SRP model behind it; your questions are directly addressed there.

by topspin 1mo ago

monocasa 1mo ago

FWIW, just having a mask in the interrupt controller is normally enough to give you the same thing at the cost of a dozen or so cycles in the critical path. Basically you just keep a mask per priority that can be built up cheaply at init time (or even compile time if you're cute about it), you apply the appropriate mask in the interrupt prologues and epilogues, and pretty much as soon as you apply the new mask in the prologue you go ahead and acknowledge the interrupt.

topspin 1mo ago
I am aware. That "dozen or so" is a problem: when everything is an interrupt, there are no interrupts: it's just scheduling, and things that must be scheduled frequently can't suffer "a dozen or so" overhead. For the SRP model to really hum, you need the silicon that solves this.
- monocasa 1mo ago
  I've found that it doesn't matter except for something that you want at the absolute highest priority anyway, which then by definition doesn't need to jump through the same hoops because nothing can preempt it anyway.
  - topspin 27d ago
    It's more about frequency than priority. When something has to be serviced tens of thousands of times a second, "a dozen or so" becomes a problem. If you have the silicon that solves this, you can retain the model. If you don't then you have to resort to workarounds. Fortunately we typically have more efficient means to deal with such hardware, but not always, and it would be a shame to break the intended model when this is the case.
amluto 1mo ago
You can do this on x86 as well at a cost a merely tens to hundreds (possibly lots of hundreds) of thousands of cycles. This is part of why x86 is so popular in the embedded space. (I’m being sarcastic, obviously. x86 interrupts and interrupt returns are hilariously slow. FRED may improve this by quite a bit.)
- nomel 1mo ago
  What's the technical reason for them being slow? Book keeping with caches or something?
  - monocasa 1mo ago
    Mostly tons of speculative state that needs to be unwound, combined with spectre mitigations, plus tons of committed state that the interrupt prologue needs to save, plus a huge song and dance to do that correctly (that FRED should help with). All combined with the fact that there's a good chance the memory the interrupt handler is going to touch isn't in the cached working set anymore, both in the actual L* caches and in subtler places like the branch predictors and TLBs.
    - rkagerer 1mo ago
      Thanks. Surely this was a performance hit even before Spectre? Also, any good technical resources that concisely describe FRED? I found this, which isn't bad but it's a bit more dumbed down than I'd like: https://www.tomshardware.com/pc-components/cpus/amd-adopts-fred-together-with-intel-for-zen-6-architecture-replacement-for-decades-old-idt-can-improve-performance-and-stability https://www.tomshardware.com/pc-components/cpus/amd-adopts-f...
    - amluto 1mo ago
      You’re missing the big ones: both the interrupt delivery and the IRET (interrupt return) mechanisms use incredibly complicated data structures to determine what the new state should be. They need to dig around in the IDT, the GDT, the TSS and possibly the LDT to find all the register values they need to set, and they need to handle all kinds of backwards compatibility. And they “serialize”, which is an extra heavyweight fence, although that only likely accounts for a few hundred cycles in each direction. Check out the pseudocode in the SDM — there are pages of it, and the pseudocode isn’t even complete. FRED simplifies the state transitions such that the new state is mostly a foregone conclusion based on MSR contents.

jacquesm 1mo ago

> One approach is to do everything in ISRs, a la RTIC. That only works for really simple systems. On more complex systems there is a pretty good chance you will end up with locked up hardware if your ISR is long enough. Interrupts need servicing to keep the data flowing, prioritization is a job for the OS, not the hardware.

monocasa 1mo ago
The model works pretty well up to much larger systems than you'd expect. If a particular interrupt has a hard real time constraint, it sounds like a great candidate for a higher priority interrupt which will let it meet that requirement. The biggest constraint is that this is really a single core model. You need something different if you go to SMP. Though there, AMP where the main core runs this 'interrrupt controller is your scheduler' scheme, and the other cores run against a work stealing scheduler for compute bound work items still is a very nice system to program against.
crest 1mo ago
The ARM v7M architecture helps a lot here because the NVIC supports priority inheritance without disabling interrupts. Yes you have to assign your priorities correctly and yield CPU time for anything compute bound, but again the architecture helps you in the form of the SVCall+PendSV exception pair. The PendSV handler can easily be multiplexed on the Cortex-M3/M4 using atomic operations on bit-banded memory. I've needed only a few dozen lines of Forth to set this up on bare metal without an RTOS to split peripheral drivers in a short interrupt handler saving the required state and delegating protocol handling to a lower priority callback.
topspin 1mo ago
> On more complex systems there is a pretty good chance you will end up with locked up hardware if your ISR is long enough. That is true of all such systems, from MCUs to the greatest CPUs ever made, and all prevailing software stacks running on them. Nothing about RTIC precludes servicing interrupts in a timely manner. It is based on a mathematical model of concurrency called Stack Resource Policy (SRP) and is entirely capable of reliably implementing whatever interrupt regime you imagine your "complex" system requires, with zero risk of priority inversion. > prioritization is a job for the OS, not the hardware So say you. The long and storied history of fragility and failure related to scheduling and interrupt handling suggests there is opportunity for greater rigor than the conventional muddle you assert as best.

mrheosuper 1mo ago

How would it handle kernel/user space if everything runs inside ISR context ?

topspin 1mo ago
Whatever OS/applications you have run in low priority ISR contexts, taking however much time they wish. When a high priority interrupt happens, the lower priority ISR contexts are preempted and then resumed after the high priority ISR is complete. Go have a look at the design and the SRP model behind it; your questions are directly addressed there.