4 ms·

I love these so much. kens: were you able to identify the original bug? I understand the errata well enough, it implies that the processor state is indetermin

by ajross 4y ago

I love these so much. kens: were you able to identify the original bug? I understand the errata well enough, it implies that the processor state is indeterminate between the segment assignment and subsequent update to SP. But this isn't a 2/386 with segment selectors; on the 8086, SS and SP are just plain registers. At least conceptually, whenever you access memory the processor does an implicit shift/add with the relevant segment; there's no intermediate state. Unless there is? Was there a special optimization that pre-baked the segment offset into SP, maybe? Would love to hear about anything you saw.

wbl 4y ago
It's not that the state is indeterminate it's that a programmer would have some serious trouble ensuring that it was possible to dump registers at that moment.
- ajross 4y ago
  No, that would just be a software bug (and indeed, interrupt handlers absolutely can't just arbitrarily trust segment state on 8086 code where app code messes with it, for exactly that reason). The errata is a hardware error: the specified behavior of the CPU after a segment assignment is clear per the docs, but the behavior of the actual CPU is apparently different.
  - mjw1007 4y ago
    The 8086 didn't have a separate stack for interrupts. So, as I understand it, the problem isn't that code running in the interrupt handler might push data to a bad place; the problem is that the the processor itself would do so when pushing the return address before branching to the interrupt handler.
    - cesarb 4y ago
      > the problem is that the the processor itself would do so when pushing the return address before branching to the interrupt handler. I want to emphasize this point: when handling an interrupt, the x86 processor itself pushes the return address and the flags into the stack. Other processor architectures store the return address and the processor state word into dedicated registers instead, and it's code running in the interrupt handler which saves the state in the stack; that would easily allow fixing the problem in software, instead of requiring it to be fixed in hardware (a simple software fix would be to use a dedicated save area or interrupt stack, separate from the normal stack; IIRC, it's that approach which later x86 processors followed, with a separate interrupt stack).
      - robocat 4y ago
        > with a separate interrupt stack Some detail here: https://www.kernel.org/doc/html/latest/x86/kernel-stacks.html https://www.kernel.org/doc/html/latest/x86/kernel-stacks.htm...
  - anyfoo 4y ago
    The CPU itself pushes flags and return address on the stack during an interrupt. If it's an NMI, you can't prevent an interrupt between mov/pop ss and mov/pop sp, which most systems will always have a need to do eventually. Therefore, the CPU has consistent state, but an extra fix was still necessary just to prevent interrupts after a mov/pop ss.
kens 4y ago
I believe this was a design bug: the hardware followed the design but nobody realized that the design had a problem. Specifically, it takes two instructions to move the stack pointer to a different segment (updating the SS and the SP). If you get an interrupt between these two instructions, everything is deterministic and "works". The problem is that the combination of the old SP and the new SS points to an unexpected place in memory, so your stack frame is going to clobber something. The hardware is doing the "right thing" but the behavior is unusable.
- ajross 4y ago
  Huh... well that's disappointing. That doesn't sound like a hardware bug to me at all. There's nothing unhandleable about this circumstance that I see, any combination of SS/SP points to a "valid" address, code just has to be sure the combination always points to enough memory to store an interrupt frame. In the overwhelmingly common situation where your stack started with an SP of 0, you just assign SP back to 0 before setting SS and nothing can go wrong (you are, after all, about to move off this stack and don't care about its contents). Even the weirdest setups can be handled with a quick thunk through an SP that overlaps in the same 64k region as the target. It's a clumsy interface but hardly something that Intel has traditionally shied away from (I mean, good grief, fast forward a half decade and they'll have inflicted protected mode transitions on us). I'm just surprised they felt this was worth a hardware patch. I guess I was hoping for something juicier.
  - pm215 4y ago
    The problem is that in general "code" can't do that because the code that's switching stack doesn't necessarily conceptually control the stack it's switching to/from. One SS:SP might point to a userspace process, for instance, and the other SS:SP be for the kernel. Both stacks are "live" in that they still have data that will later be context switched back to. You might be able to come up with a convoluted workaround, but since "switch between stacks" was a design goal (and probably assumed by various OS designs they were hoping to get ports for), the race that meant you can't do it safely was definitely a design bug.
  - 4y ago
duskwuff 4y ago
> I understand the errata well enough, it implies that the processor state is indeterminate between the segment assignment and subsequent update to SP. The root cause was effectively a bug in the programming model, not in the implementation -- there was no way to atomically relocate the stack (by updating SS and SP). The processor state wasn't indeterminate between the two instructions that updated SS and SP, but the behavior resulting from an interrupt at that point would almost certainly be unintended, since the CPU would push state onto a stack in the wrong location, potentially overwriting other stack frames, and the architecture as originally designed provided no way to prevent this.

by ajross 4y ago

wbl 4y ago

It's not that the state is indeterminate it's that a programmer would have some serious trouble ensuring that it was possible to dump registers at that moment.

ajross 4y ago
No, that would just be a software bug (and indeed, interrupt handlers absolutely can't just arbitrarily trust segment state on 8086 code where app code messes with it, for exactly that reason). The errata is a hardware error: the specified behavior of the CPU after a segment assignment is clear per the docs, but the behavior of the actual CPU is apparently different.
- mjw1007 4y ago
  The 8086 didn't have a separate stack for interrupts. So, as I understand it, the problem isn't that code running in the interrupt handler might push data to a bad place; the problem is that the the processor itself would do so when pushing the return address before branching to the interrupt handler.
  - cesarb 4y ago
    > the problem is that the the processor itself would do so when pushing the return address before branching to the interrupt handler. I want to emphasize this point: when handling an interrupt, the x86 processor itself pushes the return address and the flags into the stack. Other processor architectures store the return address and the processor state word into dedicated registers instead, and it's code running in the interrupt handler which saves the state in the stack; that would easily allow fixing the problem in software, instead of requiring it to be fixed in hardware (a simple software fix would be to use a dedicated save area or interrupt stack, separate from the normal stack; IIRC, it's that approach which later x86 processors followed, with a separate interrupt stack).
    - robocat 4y ago
      > with a separate interrupt stack Some detail here: https://www.kernel.org/doc/html/latest/x86/kernel-stacks.html https://www.kernel.org/doc/html/latest/x86/kernel-stacks.htm...
- anyfoo 4y ago
  The CPU itself pushes flags and return address on the stack during an interrupt. If it's an NMI, you can't prevent an interrupt between mov/pop ss and mov/pop sp, which most systems will always have a need to do eventually. Therefore, the CPU has consistent state, but an extra fix was still necessary just to prevent interrupts after a mov/pop ss.

kens 4y ago

I believe this was a design bug: the hardware followed the design but nobody realized that the design had a problem. Specifically, it takes two instructions to move the stack pointer to a different segment (updating the SS and the SP). If you get an interrupt between these two instructions, everything is deterministic and "works". The problem is that the combination of the old SP and the new SS points to an unexpected place in memory, so your stack frame is going to clobber something. The hardware is doing the "right thing" but the behavior is unusable.

ajross 4y ago
Huh... well that's disappointing. That doesn't sound like a hardware bug to me at all. There's nothing unhandleable about this circumstance that I see, any combination of SS/SP points to a "valid" address, code just has to be sure the combination always points to enough memory to store an interrupt frame. In the overwhelmingly common situation where your stack started with an SP of 0, you just assign SP back to 0 before setting SS and nothing can go wrong (you are, after all, about to move off this stack and don't care about its contents). Even the weirdest setups can be handled with a quick thunk through an SP that overlaps in the same 64k region as the target. It's a clumsy interface but hardly something that Intel has traditionally shied away from (I mean, good grief, fast forward a half decade and they'll have inflicted protected mode transitions on us). I'm just surprised they felt this was worth a hardware patch. I guess I was hoping for something juicier.
- pm215 4y ago
  The problem is that in general "code" can't do that because the code that's switching stack doesn't necessarily conceptually control the stack it's switching to/from. One SS:SP might point to a userspace process, for instance, and the other SS:SP be for the kernel. Both stacks are "live" in that they still have data that will later be context switched back to. You might be able to come up with a convoluted workaround, but since "switch between stacks" was a design goal (and probably assumed by various OS designs they were hoping to get ports for), the race that meant you can't do it safely was definitely a design bug.
- 4y ago

duskwuff 4y ago

> I understand the errata well enough, it implies that the processor state is indeterminate between the segment assignment and subsequent update to SP. The root cause was effectively a bug in the programming model, not in the implementation -- there was no way to atomically relocate the stack (by updating SS and SP). The processor state wasn't indeterminate between the two instructions that updated SS and SP, but the behavior resulting from an interrupt at that point would almost certainly be unintended, since the CPU would push state onto a stack in the wrong location, potentially overwriting other stack frames, and the architecture as originally designed provided no way to prevent this.