5 ms·

> IPIs (inter-processor interrupts) are inherently and hugely expensive I’d be interested to hear more about this.

pizlonator 2y ago
Let me see if I can remember what Arvind said. This was like 20 years ago so I won’t get it exactly right. He was comparing to two CPUs sharing data with shared memory. In the shared memory case, each CPU stays on the happy path but sometimes has to do some cache coherence, which may stall the CPU, but that’s the worst that can happen, generally. But if you try to send a message to another CPU, then you’re asking it to raise an interrupt. That’s not the happy path. CPU will have to stop everything to receive the interrupt and then divert execution to some interrupt gate. Arvind’s point was that every impl of this is going to be much much worse than the worst case of cache coherence. Every measurement I’ve ever made confirms this.
- keith_analog 2y ago
  There's a fairly recent study looking into the cost of raising an interrupt. TLDR: the cost on conventional systems is quite high; a significant part of the cost can be eliminated by using a custom kernel; even with a custom kernel, there remains a substantial cost. https://par.nsf.gov/servlets/purl/10079614 https://par.nsf.gov/servlets/purl/10079614 To my knowledge, the remaining cost could be decreased to approximately the same cost as a branch mispredict, but getting there would require changes to the chip hardware and software stack.
  - Dylan16807 2y ago
    > To my knowledge, the remaining cost could be decreased to approximately the same cost as a branch mispredict, but getting there would require changes to the chip hardware and software stack. Do it even need to be a misprediction? If you are completely focused on latency then flushing everything else makes sense. But I would think that if you continue execution for now and put a branch instruction into the queue you'd reduce the cost per interrupt even further.
- nirushiv 2y ago
  Sorry for the naive question - but what is an example of “send a message to another CPU”? In high level programming languages, isn’t this usually over shared memory as well? Are there constructs to directly “message” a CPU? When might one use it?
- Dylan16807 2y ago
  Most processors are designed with very expensive task switching, but I don't think I'd call that "inherent". A processor could implement an interrupt as a forced branch 200 cycles in the future. Then it's the same speed as a function call with callee-save registers. That's still slower than letting cache hardware manage the whole thing in the background, but it should be as little as nanoseconds slower.

Let me see if I can remember what Arvind said. This was like 20 years ago so I won’t get it exactly right. He was comparing to two CPUs sharing data with shared memory. In the shared memory case, each CPU stays on the happy path but sometimes has to do some cache coherence, which may stall the CPU, but that’s the worst that can happen, generally. But if you try to send a message to another CPU, then you’re asking it to raise an interrupt. That’s not the happy path. CPU will have to stop everything to receive the interrupt and then divert execution to some interrupt gate. Arvind’s point was that every impl of this is going to be much much worse than the worst case of cache coherence. Every measurement I’ve ever made confirms this.

keith_analog 2y ago
There's a fairly recent study looking into the cost of raising an interrupt. TLDR: the cost on conventional systems is quite high; a significant part of the cost can be eliminated by using a custom kernel; even with a custom kernel, there remains a substantial cost. https://par.nsf.gov/servlets/purl/10079614 https://par.nsf.gov/servlets/purl/10079614 To my knowledge, the remaining cost could be decreased to approximately the same cost as a branch mispredict, but getting there would require changes to the chip hardware and software stack.
- Dylan16807 2y ago
  > To my knowledge, the remaining cost could be decreased to approximately the same cost as a branch mispredict, but getting there would require changes to the chip hardware and software stack. Do it even need to be a misprediction? If you are completely focused on latency then flushing everything else makes sense. But I would think that if you continue execution for now and put a branch instruction into the queue you'd reduce the cost per interrupt even further.
nirushiv 2y ago
Sorry for the naive question - but what is an example of “send a message to another CPU”? In high level programming languages, isn’t this usually over shared memory as well? Are there constructs to directly “message” a CPU? When might one use it?
Dylan16807 2y ago
Most processors are designed with very expensive task switching, but I don't think I'd call that "inherent". A processor could implement an interrupt as a forced branch 200 cycles in the future. Then it's the same speed as a function call with callee-save registers. That's still slower than letting cache hardware manage the whole thing in the background, but it should be as little as nanoseconds slower.