5 ms·

meh, i'm an idiot.

jsnell 9y ago
The question isn't about three instructions being executed by three separate OS threads. It's about three instructions being executed at the same time on a single thread on a single core. The interrupt context will only contain a single program counter value; which of the instructions that could currently be considered to be executing on that thread will it actually point to?
- rootw0rm 9y ago
  i figured i misunderstood. i guess if you run it long enough that statistically it doesn't matter? i still think the culprit is going to turn out to be something more mundane.
- tedunangst 9y ago
  I would guess PC is most recently retired instruction? Or would it be not retired? Although that is an interesting scenario. slow eax, ebx fast ecx, ecx Assuming slow and fast always finish at the same time, will the profiler ever see the CPU executing slow?
  - comex 9y ago
    The PC would be recorded from an interrupt handler as the address to resume execution at; I don’t think there’s any other way to record PCs on x86 (there’s Last Branch Record but that’s only for branch instructions). So, I’d expect it would be the first not-yet-retired instruction, and you’d always see ‘slow’, not ‘fast’.
    - brucedawson 9y ago
      > you’d always see ‘slow’, not ‘fast’. And yet - line 34 of the spreadsheet shows "xor r8d,r8d" as being quite expensive. My theory is that the branch which targets it must be mispredicted and it gets blamed. Similarly, line 24 of the spreadsheet shows "cmp ebx,eax" as being quite expensive. The only explanation I can come up with is that the previous instruction was blocked by a cache miss and then the "cmp" must have been folded in with the add so that mumble mumble nope I don't have a good mental model. So, there are quite a few examples of instructions which by themselves are obviously always fast being hit by a huge portion of the CPU samples.
  - jsnell 9y ago
    Working on the most recently retired instruction would be odd, since it'd lead to things like the cost of cache misses being assigned to instructions that have nothing to do with memory access. But other than that, I've never been able to build a mental model for exactly how the accounting works. For example take an inner loop with no memory accesses, like this: 18: add $0x1,%rax cmp %rax,%rdx jne 18 On a Skylake this will execute once per cycle (IPC of ~3), so we're clearly spending equal amounts of time on each instruction. But perf will assign basically all of the cost to the jump. 0.28 │18:┌─→add $0x1,%rax 0.27 │ │ cmp %rax,%rdx 99.45 │ └──jne 18 So it could be consistent with some kind of "first in-flight" or "last in-flight" model. But change the loop to be 5 instructions long (and with an IPC of ~5, so still executing the whole loop once per cycle), and suddenly the PC gets smeared across two instructions: 2.12 │20:┌─→add %rsi,%rax 0.94 │ │ add $0x1,%rcx 45.36 │ │ add %rax,%rdx 0.10 │ │ cmp %rcx,%rsi 51.48 │ └──jne 20 Take an almost equivalent loop, just swapping two instructions. It's still 5 instructions and an IPC of ~5. And... 80.79 │20: add $0x1,%rcx 1.29 │ add %rax,%rdx 0.89 │ add %rsi,%rax 0.00 │ cmp %rcx,%rsi 17.02 │ ↑ jne 20 Multiple runs of an individual test case will be pretty consistent on a specific machine; so it's not that the assignment is just random. It's deterministic, but the exact mechanism is very opaque.
    - tedunangst 9y ago
      Oh, nice examples. (And yeah, had I thought another minute, the PC will never be actually retired, but something after.)
    - ot 9y ago
      > it'd lead to things like the cost of cache misses being assigned to instructions that have nothing to do with memory access. But this happens all the time with perf. I often see cache misses attributed to a jump dependent on a data load.
      - jsnell 9y ago
        Sure, but that's still an in-flight instruction that is stalled waiting on a memory access (just indirectly). You don't see the miss being attributed to the instructions that preceded the load.
  - pjc50 9y ago
    https://stackoverflow.com/questions/8902132/when-an-interrupt-occurs-what-happens-to-instructions-in-the-pipeline https://stackoverflow.com/questions/8902132/when-an-interrup... has some discussion. Basically the priority of the Intel design (and most successful processors) is to preserve the illusion of a series of instructions executing one after the other. So the control logic has to keep track of the "waterline", the point before which all instructions have been completely committed and retired. Everything beyond there is aborted on an interrupt.

meh, i'm an idiot.

The question isn't about three instructions being executed by three separate OS threads. It's about three instructions being executed at the same time on a single thread on a single core. The interrupt context will only contain a single program counter value; which of the instructions that could currently be considered to be executing on that thread will it actually point to?

rootw0rm 9y ago
i figured i misunderstood. i guess if you run it long enough that statistically it doesn't matter? i still think the culprit is going to turn out to be something more mundane.
tedunangst 9y ago
I would guess PC is most recently retired instruction? Or would it be not retired? Although that is an interesting scenario. slow eax, ebx fast ecx, ecx Assuming slow and fast always finish at the same time, will the profiler ever see the CPU executing slow?
- comex 9y ago
  The PC would be recorded from an interrupt handler as the address to resume execution at; I don’t think there’s any other way to record PCs on x86 (there’s Last Branch Record but that’s only for branch instructions). So, I’d expect it would be the first not-yet-retired instruction, and you’d always see ‘slow’, not ‘fast’.
  - brucedawson 9y ago
    > you’d always see ‘slow’, not ‘fast’. And yet - line 34 of the spreadsheet shows "xor r8d,r8d" as being quite expensive. My theory is that the branch which targets it must be mispredicted and it gets blamed. Similarly, line 24 of the spreadsheet shows "cmp ebx,eax" as being quite expensive. The only explanation I can come up with is that the previous instruction was blocked by a cache miss and then the "cmp" must have been folded in with the add so that mumble mumble nope I don't have a good mental model. So, there are quite a few examples of instructions which by themselves are obviously always fast being hit by a huge portion of the CPU samples.
- jsnell 9y ago
  Working on the most recently retired instruction would be odd, since it'd lead to things like the cost of cache misses being assigned to instructions that have nothing to do with memory access. But other than that, I've never been able to build a mental model for exactly how the accounting works. For example take an inner loop with no memory accesses, like this: 18: add $0x1,%rax cmp %rax,%rdx jne 18 On a Skylake this will execute once per cycle (IPC of ~3), so we're clearly spending equal amounts of time on each instruction. But perf will assign basically all of the cost to the jump. 0.28 │18:┌─→add $0x1,%rax 0.27 │ │ cmp %rax,%rdx 99.45 │ └──jne 18 So it could be consistent with some kind of "first in-flight" or "last in-flight" model. But change the loop to be 5 instructions long (and with an IPC of ~5, so still executing the whole loop once per cycle), and suddenly the PC gets smeared across two instructions: 2.12 │20:┌─→add %rsi,%rax 0.94 │ │ add $0x1,%rcx 45.36 │ │ add %rax,%rdx 0.10 │ │ cmp %rcx,%rsi 51.48 │ └──jne 20 Take an almost equivalent loop, just swapping two instructions. It's still 5 instructions and an IPC of ~5. And... 80.79 │20: add $0x1,%rcx 1.29 │ add %rax,%rdx 0.89 │ add %rsi,%rax 0.00 │ cmp %rcx,%rsi 17.02 │ ↑ jne 20 Multiple runs of an individual test case will be pretty consistent on a specific machine; so it's not that the assignment is just random. It's deterministic, but the exact mechanism is very opaque.
  - tedunangst 9y ago
    Oh, nice examples. (And yeah, had I thought another minute, the PC will never be actually retired, but something after.)
  - ot 9y ago
    > it'd lead to things like the cost of cache misses being assigned to instructions that have nothing to do with memory access. But this happens all the time with perf. I often see cache misses attributed to a jump dependent on a data load.
    - jsnell 9y ago
      Sure, but that's still an in-flight instruction that is stalled waiting on a memory access (just indirectly). You don't see the miss being attributed to the instructions that preceded the load.
- pjc50 9y ago
  https://stackoverflow.com/questions/8902132/when-an-interrupt-occurs-what-happens-to-instructions-in-the-pipeline https://stackoverflow.com/questions/8902132/when-an-interrup... has some discussion. Basically the priority of the Intel design (and most successful processors) is to preserve the illusion of a series of instructions executing one after the other. So the control logic has to keep track of the "waterline", the point before which all instructions have been completely committed and retired. Everything beyond there is aborted on an interrupt.