6 ms·

It still wouldn't be reliable because if a kernel is fully wedged its not going to respond to anything, including an NMI. I suspect that is why you are seeing

by throwaway2048 7y ago

It still wouldn't be reliable because if a kernel is fully wedged its not going to respond to anything, including an NMI. I suspect that is why you are seeing unreliability with unknown NMI panics.

amluto 7y ago
No it’s not. If you have a legit source of NMIs (e.g. if you’re running perf), it’s entirely possible for a magic diagnostic NMI to be ignored because it is I distinguishable from a perf NMI.
derefr 7y ago
You can't be so wedged that you can't respond to an NMI; an NMI is the CPU stealing execution from the kernel and giving it to the configured interrupt handler. The interrupt handler is usually coded entirely independently of any state in the kernel itself, so—for any reasonable production kernel—there's nothing the kernel can do that would make the NMI interrupt handler fail to function.
- amluto 7y ago
  Not really. An NMI is delivered just like any other interrupt, and the NMI handler absolutely depends on kernel state. This is needed for the handler to do things like figuring out that it's a performance counter event and queueing up the perf event record involved. There are a few relevant things about interrupts that matter here: 1. Interrupts, in general, don't have useful payloads. This is because the CPU and its interrupt controller keep track of all interrupts pending so that they can make sure to invoke the interrupt handler when it's next appropriate to do so. To avoid having a potentially unbounded queue of pending interrupts, the pending interrupt state is just a mask of bits indicating which vectors are pending. On x86, NMI is vector 2. It is pending or it isn't. This is like like a regular device IRQ, which can be pending or not. In the case of regular device IRQs, the OS can query the APIC to get more information. If the IRQ vector is shared, the OS can query all possible sources. In the case of performance counter NMIs, the kernel can read the performance counters. In the case of magic AWS NMIs, there is nothing to query. This is why I think it's a poor design. 2. The x86 NMI design is problematic: NMIs are blocked while the CPU thinks an NMI is running, but the kernel has insufficient control of the "NMIs are blocked" bit, and the CPUs heuristic for "is an NMI is running?" is inaccurate. The result is extremely nasty asm code to fix things up after the CPU messes them up. This is nasty but manageable. 3. x86_64 has some design errors that mean that the kernel must occasionally run with an invalid stack pointer. The kernel developers have no choice in the matter. Regular interrupts are sensibly masked when this happens, so interrupt delivery won't explode when the stack pointer is garbage. But NMIs are non-maskable by design, which necessitates a series of unpleasant workarounds that, in turn, cause their own problems. The kernel can definitely have bugs that cause NMI handling to fail. I've found and fixed quite a few over the years. Fortunately, no amount of memory pressure, infinite looping, or otherwise getting stuck outside the NMI code is likely to prevent NMI delivery. What will kill AWS's clever idea is if the kernel holds locks needed to create a stackdump at the time that the NMI is delivered. The crashkernel mechanism tries to work around this, but I doubt it's perfect.
  - mrbrowning 7y ago
    If you're able to, would you mind linking to some resources about the case you describe in point three where the kernel need deal with an invalid stack pointer? I'm very curious about the underlying causes. Also, regarding point two, is the cumulative result of those factors you describe (other NMIs being blocked, lack of kernel control, the unreliable heuristic) that a diagnostic NMI may end up never being run?
    - amluto 7y ago
      > If you're able to, would you mind linking to some resources about the case you describe in point three where the kernel need deal with an invalid stack pointer? Look up SYSCALL in the manual (AMD APM or Intel SDM). SYSCALL switches to kernel mode without changing RSP at all. This means that at least the first few instructions of the kernel’s SYSCALL entry point have a completely bogus RSP. An NMI, MCE, or #DB hitting there is fun. For the latter, see CVE-2018-8897. You can also read the actual NMI entry asm in Linux’s arch/x86/entry/entry_64.S for how the kernel handles an NMI while trying to return from an NMI :) To some extent, the x86_64 architecture is a pile of kludges all on top of each other. Somehow it all mostly works. > is the cumulative result of those factors you describe (other NMIs being blocked, lack of kernel control, the unreliable heuristic) that a diagnostic NMI may end up never being run? No, that’s unrelated. A diagnostic NMI causes the kernel’s NMI vector to be invoked, and that’s all. Suppose that this happens concurrently with a perf NMI. There could be no indication whatsoever that the diagnostic NMI happened: the two NMIs can get coalesced, and, as far as the kernel can tell, only the perf NMI happened. Once all the weird architectural junk is out of the way, the NMI handler boils down to: for each possible NMI cause did it happen? if so, handle it. If no cause was found, complain. Amazon’s thing is trying to hit the “complain” part. What they should do is give some readable indication that it happened so it can be added to the list of possible causes.
- anyfoo 7y ago
  So if the kernel loads a random value into CR3, will an NMI still do anything useful? (Or if the IDT is overwritten, or the NMI handler unmapped.) I don't know if that's enough to make an NMI "unreliable", but the kernel being able to do nothing for NMIs to fail might be a bit strong.
  - amluto 7y ago
    Total garbage in CR3 will likely triple fault. Garbage in the IDT, etc will also double or triple fault. Corruption of the NMI “don’t recurse now” state will likely just ignore the NMI. Corruption of various other things may cause the NMI handler to crash before it makes a useful dump. A triple fault will reboot or do whatever else the hypervisor feels like doing.
    - anyfoo 7y ago
      Yeah, that was my point. No kernel panic on a triple fault. Though arguably, a hypervisor can dump out some useful state on a triple fault, as they actually tend to do. But that's not an in-band kernel panic, and in VirtualBox at least you can get a dump of that state without sending an NMI or otherwise disturbing the VM.

It still wouldn't be reliable because if a kernel is fully wedged its not going to respond to anything, including an NMI. I suspect that is why you are seeing unreliability with unknown NMI panics.

amluto 7y ago

No it’s not. If you have a legit source of NMIs (e.g. if you’re running perf), it’s entirely possible for a magic diagnostic NMI to be ignored because it is I distinguishable from a perf NMI.

derefr 7y ago

You can't be so wedged that you can't respond to an NMI; an NMI is the CPU stealing execution from the kernel and giving it to the configured interrupt handler. The interrupt handler is usually coded entirely independently of any state in the kernel itself, so—for any reasonable production kernel—there's nothing the kernel can do that would make the NMI interrupt handler fail to function.

amluto 7y ago
Not really. An NMI is delivered just like any other interrupt, and the NMI handler absolutely depends on kernel state. This is needed for the handler to do things like figuring out that it's a performance counter event and queueing up the perf event record involved. There are a few relevant things about interrupts that matter here: 1. Interrupts, in general, don't have useful payloads. This is because the CPU and its interrupt controller keep track of all interrupts pending so that they can make sure to invoke the interrupt handler when it's next appropriate to do so. To avoid having a potentially unbounded queue of pending interrupts, the pending interrupt state is just a mask of bits indicating which vectors are pending. On x86, NMI is vector 2. It is pending or it isn't. This is like like a regular device IRQ, which can be pending or not. In the case of regular device IRQs, the OS can query the APIC to get more information. If the IRQ vector is shared, the OS can query all possible sources. In the case of performance counter NMIs, the kernel can read the performance counters. In the case of magic AWS NMIs, there is nothing to query. This is why I think it's a poor design. 2. The x86 NMI design is problematic: NMIs are blocked while the CPU thinks an NMI is running, but the kernel has insufficient control of the "NMIs are blocked" bit, and the CPUs heuristic for "is an NMI is running?" is inaccurate. The result is extremely nasty asm code to fix things up after the CPU messes them up. This is nasty but manageable. 3. x86_64 has some design errors that mean that the kernel must occasionally run with an invalid stack pointer. The kernel developers have no choice in the matter. Regular interrupts are sensibly masked when this happens, so interrupt delivery won't explode when the stack pointer is garbage. But NMIs are non-maskable by design, which necessitates a series of unpleasant workarounds that, in turn, cause their own problems. The kernel can definitely have bugs that cause NMI handling to fail. I've found and fixed quite a few over the years. Fortunately, no amount of memory pressure, infinite looping, or otherwise getting stuck outside the NMI code is likely to prevent NMI delivery. What will kill AWS's clever idea is if the kernel holds locks needed to create a stackdump at the time that the NMI is delivered. The crashkernel mechanism tries to work around this, but I doubt it's perfect.
- mrbrowning 7y ago
  If you're able to, would you mind linking to some resources about the case you describe in point three where the kernel need deal with an invalid stack pointer? I'm very curious about the underlying causes. Also, regarding point two, is the cumulative result of those factors you describe (other NMIs being blocked, lack of kernel control, the unreliable heuristic) that a diagnostic NMI may end up never being run?
  - amluto 7y ago
    > If you're able to, would you mind linking to some resources about the case you describe in point three where the kernel need deal with an invalid stack pointer? Look up SYSCALL in the manual (AMD APM or Intel SDM). SYSCALL switches to kernel mode without changing RSP at all. This means that at least the first few instructions of the kernel’s SYSCALL entry point have a completely bogus RSP. An NMI, MCE, or #DB hitting there is fun. For the latter, see CVE-2018-8897. You can also read the actual NMI entry asm in Linux’s arch/x86/entry/entry_64.S for how the kernel handles an NMI while trying to return from an NMI :) To some extent, the x86_64 architecture is a pile of kludges all on top of each other. Somehow it all mostly works. > is the cumulative result of those factors you describe (other NMIs being blocked, lack of kernel control, the unreliable heuristic) that a diagnostic NMI may end up never being run? No, that’s unrelated. A diagnostic NMI causes the kernel’s NMI vector to be invoked, and that’s all. Suppose that this happens concurrently with a perf NMI. There could be no indication whatsoever that the diagnostic NMI happened: the two NMIs can get coalesced, and, as far as the kernel can tell, only the perf NMI happened. Once all the weird architectural junk is out of the way, the NMI handler boils down to: for each possible NMI cause did it happen? if so, handle it. If no cause was found, complain. Amazon’s thing is trying to hit the “complain” part. What they should do is give some readable indication that it happened so it can be added to the list of possible causes.
anyfoo 7y ago
So if the kernel loads a random value into CR3, will an NMI still do anything useful? (Or if the IDT is overwritten, or the NMI handler unmapped.) I don't know if that's enough to make an NMI "unreliable", but the kernel being able to do nothing for NMIs to fail might be a bit strong.
- amluto 7y ago
  Total garbage in CR3 will likely triple fault. Garbage in the IDT, etc will also double or triple fault. Corruption of the NMI “don’t recurse now” state will likely just ignore the NMI. Corruption of various other things may cause the NMI handler to crash before it makes a useful dump. A triple fault will reboot or do whatever else the hypervisor feels like doing.
  - anyfoo 7y ago
    Yeah, that was my point. No kernel panic on a triple fault. Though arguably, a hypervisor can dump out some useful state on a triple fault, as they actually tend to do. But that's not an in-band kernel panic, and in VirtualBox at least you can get a dump of that state without sending an NMI or otherwise disturbing the VM.