3 ms·

Interesting walk through in the linked video as he tried to troubleshoot this: https://www.youtube.com/live/1UnoBfw6soI https://www.youtube.com/live/1UnoBfw6so

Interesting walk through in the linked video as he tried to troubleshoot this: https://www.youtube.com/live/1UnoBfw6soI https://www.youtube.com/live/1UnoBfw6soI Pretty shocking to see (extremely unlikely) non-malicious code work / not-work depending on a security mitigation setting. Curious to see where this goes as to whether it's a kernel bug and nobody is paying attention to `mitigations=off` now or the unlikely outcome that it's an actual hardware bug where mitigations work around it and nobody has noticed. Seems he had overclocked and maybe disabling the migation has revealed an instability in his setup that's otherwise stable (or at best marginal).

yellow_lead 3y ago
He says that he tried stock voltage for everything, and even undervolted/underclocked CPU/memory and still had the bug.
tux3 3y ago
mitigations=off does a lot of things, it's a bundle of options Perhaps the next step is trying to figure out which mitigation exactly causes it to fail. Then kernel peeps should have a fighting chance of tracking this down
- simcop2387 3y ago
  In the video comments it looks like he's done some of that and it's one of the spectre v2 mitigations. Makes me wonder if there's now less QA happening on the oem side without mitigations=on.
- AnotherGoodName 3y ago
  SMT off fixing it is a good clue. It heavily points to the problem being within a core rather than between execution cores. That's still a broad area though. Within a core some state is inadvertantly shared between the shared threads running on it but there's a lot of possibilities of what that could be.
  - foota 3y ago
    Some state being in the CPU or kernel? I guess if you have per core state in the kernel that's only accessed by that core, then you're safe to modify it without locking, but this isn't the case for data shared between hyperthreads of a core?
    - AnotherGoodName 3y ago
      Within the CPU core (terminology here gets awkward). Spectre in its simplest to exploit form used the fact that the branch prediction based state wasn't completely cleared between running different threads on the CPU and led to a really easy timing attack. Time your own code to see which way the branch of the previous code likely went. There's a lot of CVE's around this but that's the simplest case. The mitigations work by making sure to clear everything they can when the CPU switches contexts. This includes clearing loaded local CPU cache, clearing the branch prediction table, clearing all speculative execution entries. The mitigations likely avoid a crash by clearing something that should be cleared between context switches in all cases. As in the mitigations likely accidentally fix this. Which is a good clue. It'd be nice if the mitigations were more fine grained than all or nothing so we could turn on the mitigations one by one but the microcode is not open so we can't do this :(.
      - vlovich123 3y ago
        Is it likely the issue is within the Linux kernel itself or within the microcode for the CPU?
        deleted 3y ago
        [deleted]
        deleted 3y ago
        [deleted]
        AnotherGoodName 3y ago
        I'm going to say microcode until someone reproduces on a completely different architecture :).
tyrfing 3y ago
Mitigations can have very large effects on system stability, I ran into this with my old Haswell system. Mitigations on allowed, I believe, 100Mhz higher at lower voltage - but it may have been 200Mhz. These settings were tested over many months, completely stable, and mitigations off with the same setting wouldn't even allow booting. Huge effect relative to anything else, basically like adding an extra 0.1V vcore.
- deleted 3y ago
  [deleted]
Levitating 3y ago
What does he say at 8:17?[1] Arsink Overism? [1]: https://www.youtube.com/live/1UnoBfw6soI?si=R8nJ1FxdE4zBuO0i&t=497 https://www.youtube.com/live/1UnoBfw6soI?si=R8nJ1FxdE4zBuO0i...
- mastensg 3y ago
  "rsync algorithm"

yellow_lead 3y ago

He says that he tried stock voltage for everything, and even undervolted/underclocked CPU/memory and still had the bug.

tux3 3y ago

mitigations=off does a lot of things, it's a bundle of options Perhaps the next step is trying to figure out which mitigation exactly causes it to fail. Then kernel peeps should have a fighting chance of tracking this down

simcop2387 3y ago
In the video comments it looks like he's done some of that and it's one of the spectre v2 mitigations. Makes me wonder if there's now less QA happening on the oem side without mitigations=on.
AnotherGoodName 3y ago
SMT off fixing it is a good clue. It heavily points to the problem being within a core rather than between execution cores. That's still a broad area though. Within a core some state is inadvertantly shared between the shared threads running on it but there's a lot of possibilities of what that could be.
- foota 3y ago
  Some state being in the CPU or kernel? I guess if you have per core state in the kernel that's only accessed by that core, then you're safe to modify it without locking, but this isn't the case for data shared between hyperthreads of a core?
  - AnotherGoodName 3y ago
    Within the CPU core (terminology here gets awkward). Spectre in its simplest to exploit form used the fact that the branch prediction based state wasn't completely cleared between running different threads on the CPU and led to a really easy timing attack. Time your own code to see which way the branch of the previous code likely went. There's a lot of CVE's around this but that's the simplest case. The mitigations work by making sure to clear everything they can when the CPU switches contexts. This includes clearing loaded local CPU cache, clearing the branch prediction table, clearing all speculative execution entries. The mitigations likely avoid a crash by clearing something that should be cleared between context switches in all cases. As in the mitigations likely accidentally fix this. Which is a good clue. It'd be nice if the mitigations were more fine grained than all or nothing so we could turn on the mitigations one by one but the microcode is not open so we can't do this :(.
    - vlovich123 3y ago
      Is it likely the issue is within the Linux kernel itself or within the microcode for the CPU?
      - deleted 3y ago
        [deleted]
      - deleted 3y ago
        [deleted]
      - AnotherGoodName 3y ago
        I'm going to say microcode until someone reproduces on a completely different architecture :).

tyrfing 3y ago

Mitigations can have very large effects on system stability, I ran into this with my old Haswell system. Mitigations on allowed, I believe, 100Mhz higher at lower voltage - but it may have been 200Mhz. These settings were tested over many months, completely stable, and mitigations off with the same setting wouldn't even allow booting. Huge effect relative to anything else, basically like adding an extra 0.1V vcore.

deleted 3y ago
[deleted]

Levitating 3y ago

What does he say at 8:17?[1] Arsink Overism? [1]: https://www.youtube.com/live/1UnoBfw6soI?si=R8nJ1FxdE4zBuO0i&t=497 https://www.youtube.com/live/1UnoBfw6soI?si=R8nJ1FxdE4zBuO0i...

mastensg 3y ago
"rsync algorithm"