11 ms·

AMD Zen2 ymm registers rolling back

277 points by taviso 4y ago

SethTro 4y ago
https://lkml.org/lkml/2023/2/22/982 https://lkml.org/lkml/2023/2/22/982 Seems to be a known errata which was fixed by a microcode update
- sp332 4y ago
  https://mobile.twitter.com/taviso/status/1630695259935219713 https://mobile.twitter.com/taviso/status/1630695259935219713 Apparently some Ryzen models have no fixed microcode available. You can boot with clearcpuid=xsaves as a workaround, probably at some performance cost.
  - stefan_ 4y ago
    As I understood the email thread, they do have microcode updates but they weren't actually released anywhere but in some crusty vendors BIOS update, so you can only get them if someone fished them out of there. i.e. thats what this repo seems to be: https://github.com/platomav/CPUMicrocodes https://github.com/platomav/CPUMicrocodes
    - deleted 4y ago
      [deleted]
    - caf 4y ago
      The person who reported the bug in their family 0x17 model 0x60 Renoir SoC said it wasn't fixed even by the latest BIOS-supplied microcode available.
      - lathiat 4y ago
        The reality is most consumer motherboards rarely post updates especially after the first year or so. You'll tend to get updates to fix CPU compatability for newer CPUs if the motherboard is still on sale, but otherwise most long term BIOS updates seem largely to be from enterprise vendors (Dell, Lenovo, etc) and much less common on consumer or gaming type hardware. I think most people rely on the operating system to (amazingly) hot patch it during boot. Intel and AMD both publish them and are integrated regularly into most distros (and the linux-firmware git tree). Surprising/weird that they haven't released the Renoir ones. Also seems Tavis had a bug where Debian wasn't applying them on boot for one reason but didn't give details. Wonder what it was.
        Semaphor 4y ago
        > The reality is most consumer motherboards rarely post updates especially after the first year or so I can’t confirm that. My current board is the MSI X570-A PRO. First BIOS was 2019-06-20, the latest 2022-08-19. And that’s still updating versions and settings. After 3 years, and I’m expecting more. This has also been my experience with other boards. MB updates tend to last several years.
        kllrnohj 4y ago
        You have that like exactly backwards. You'll get a lot fewer bios updates from Dell or Lenovo than you will from MSI, Asus, Gigabyte, etc.. consumer / gaming motherboard lines. My 5 year old X370-F GAMING is still getting BIOS updates. Others, like MSI, practically forced AMD to continue issuing AGESA updates for X370 & X470 chipsets after AMD had announced official end of support - they got AMD to change course and add new CPU support to those old chipsets. But otherwise all the major consumer / gaming motherboards pick up new AGESA updates quickly & consistently, even when they're EOL platforms.
        oynqr 4y ago
        I have a really old GA-Z87X-UD5H, it got NVME support five years after release. Probably by accident, but still.
    - jabl 4y ago
      Since this is about Linux, the microcodes it applies on boot can be found in the linux-firmware repository. For AMD microcode, in particular https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/amd-ucode https://git.kernel.org/pub/scm/linux/kernel/git/firmware/lin... However, for some unexplainable reason AMD doesn't tend to update the microcodes in that repo particularly often, leaving it up to BIOS vendors and users updating their BIOS.
  - amluto 4y ago
    No obvious reason for a performance cost. If XSAVEC is available, performance should be essentially identical to XSAVES.
  - BoardsOfCanada 4y ago
    I had to look up the difference between XSAVES and XSAVEC: "Execution of XSAVES is similar to that of XSAVEC. XSAVES differs from XSAVEC in that it can save state components corresponding to bits set in the IA32_XSS MSR and that it may use the modified optimization."
    - caf 4y ago
      In this case it seems like the "modified optimization" is where the bug lies.
- caf 4y ago
  It's erratum #1386 (see https://www.amd.com/system/files/TechDocs/56323-PUB_1.00.pdf https://www.amd.com/system/files/TechDocs/56323-PUB_1.00.pdf ): The XSAVES instruction may fail to save XMM registers to the provided state save area if all of the following are true: • All XMM registers were restored to the initialization value by the most recent XRSTORS instruction because the XSTATE_BV[SSE] bit was clear. • The state save area for the XMM registers does not contain the initialization state. • The value in the XMM registers match the initialization value when the XSAVES instruction is executed. • The MXCSR register has been modified to a value different from the initialization value since the most recent XRSTORS instruction.
- pabs3 4y ago
  Linux folks also plan to apply a workaround for systems running old microcode.
- alex_duf 4y ago
  As an outsider from the hardware world, I find it astounding that it's possible to fix the behaviour a CPU instruction by changing code. (assuming I understand correctly) In my mind a CPU instruction is hardwired on the chip, and it blows my mind that we keep finding workarounds to already released hardware. Maybe someone could dumb that down for me?
  - fhars 4y ago
    The CPU only pretends to be a CPU. In reality, it is a small datacenter comprised of several small special purpose computers doing all the work. I gave up understanding CPUs in depth by the time I read an introducion to Intel's then-new i860 CPU in the April issue of a magazine and it turned out to be a real device.
    - psychphysic 4y ago
      Best explanation in my opinion. Almost like a CPU has an JIT emulator for x86.
      - yvdriess 4y ago
        ucode cracking is relatively straightforward, I wouldn't call it a JIT. https://intelxed.github.io/ref-manual/ https://intelxed.github.io/ref-manual/ It's more like an intepreter for x86, seeing as it is actually executed on a dataflow architecture.
        psychphysic 4y ago
        Macro op fusion is more what I had in mind when I called it JIT and not interpretation. Not sure what the distinction is but that's where it is to my mind
        yvdriess 4y ago
        It's true that the distinction is a bit vague, the term JIT is overloaded to enough that it stopped being a useful technical term. Compared to 'JVM JIT' or 'LuaJIT': there is no instrumentation to detect what is hot or not. The CPU frontend will crack x86 instructions into micro-ops, while looking for some patterns to merge certain instructions or uops into larger micro-ops. The micro-coded instructions (like many of the legacy instructions) are likely just lookups. Most of this is my speculation, mind. Modern CPU frontends are still kind of a black magic box to me, but I think they are limited by relatively simple transformations by virtue of being on the critical execution path.
snvzz 4y ago
Lovely reminder of the mess x86 is. fp registers shared with mmx, sse registers (xmm), avx registers (ymm), and a truckload of them. Modern implementations have extremely complex frontends, full of elaborate hacks to get performance despite x86. Complexity breeds bugs, such as this one.
- userbinator 4y ago
  Look at the errata for typical ARM SoCs if you think x86 is bad. A lot of them aren't even publicly available.
- reisse 4y ago
  I wonder how much performance is really "despite" x86, and how much is thanks to it. To this day, x86 CPUs absolutely dominate everything in compute power. Sure, ARMs are better in performance-per-watt game, but the question of how to scale them to the level of high-end x86 desktop processors is still open. For now, I'd argue it's not even clear if that's possible.
  - nine_k 4y ago
    Aren't Apple M2 chips solid desktop-level processors based on ARM?
    - api 4y ago
      High end Intel and AMD chips hold the performance crown but the M chips utterly destroy them on performance per watt. It’s not even close. It’s a mix of a simpler ISA, good core design, and small process nodes.
      - erik 4y ago
        > It’s a mix of a simpler ISA, good core design, and small process nodes. It's also a design that prioritizes perf/watt, whereas CPU vendors tend to prioritize perf/area. (aka perf/$)
naikrovek 4y ago
tavis is everywhere in this space. respect.
deleted 4y ago
[deleted]
userbinator 4y ago
Unless you absolutely need the newest for some reason, I think buying CPUs which have been out for a year if not more would be the best way of avoiding hardware bugs like this, as it seems like they've also started using users for random spot-testing instead of verifying against a spec. One would hope they have plenty of esoteric and freely available x86/PC software to use for regression testing, like demoscene productions going back to the 80s (a great way to exercise opcode sequences that compilers might rarely or never create, but should still work), but with the recent CPU bugs, it feels more like "Windows/Linux/$common_OS with $common_software seems to work, ship it!"
- scottlamb 4y ago
  I think this is solid advice but hardly a guarantee. Note this is discussing Zen2, not Zen4. wikipedia says [1] Zen2 Launched 7 July 2019; 3 years ago. [1] https://en.wikipedia.org/wiki/Zen_2 https://en.wikipedia.org/wiki/Zen_2
  - Tuna-Fish 4y ago
    Yes, and this bug was fixed years ago. The person who stumbled upon it only did so because the correct microcode update was not being loaded. Although the fact that there apparently exist some APUs that have not had the update applied because of shitty OEMs is very concerning.
    - jabl 4y ago
      Also AMD being shitty and not publishing microcode updates to the linux-firmware repo so users would get access to them despite crappy OEM's or users neglecting to do BIOS updates.
- suresk 4y ago
  Zen 2 architecture came out in 2019 and this particular processor in 2020. I don't think your advice would work here, and there are definite downsides to waiting too many years after a processor has been released to buy it.
- 4y ago
Deeg9rie9usi 4y ago
Please don't link lkml.org, use lore instead: https://lore.kernel.org/lkml/Y%2FW4x7%2FKFqmDmmR7@thinkstation.cmpxchg8b.net/#r https://lore.kernel.org/lkml/Y%2FW4x7%2FKFqmDmmR7@thinkstati...
- hardware2win 4y ago
  It is unreadable, wtf?? Who the hell came with such a layout. Is it really this hard to copy industry standard like github?
  - oblio 4y ago
    He he. It's just copied off basic mailing list designs from the 90s (or the 80s?). That design is basically what happens when you tell a kernel dev to write a mailing list archiving UI. It's clean because they value simplicity, but they obviously have ~0 non-CLI UI/UX experience.
    - Deeg9rie9usi 4y ago
      Since you are an UI/UX expert, send a patch to https://repo.or.cz/public-inbox.git https://repo.or.cz/public-inbox.git
      - oblio 4y ago
        Sending patches by mail is also bad UI/UX :-p
- VMG 4y ago
  why?
  - Deeg9rie9usi 4y ago
    lkml.org was ditched some time ago by kernel devs. It had often issues, is cropping messages, has no export function, has sometimes even ads on the site. That's why kernel.org folks run now their own reliable archive. See: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=05a5f51ca566 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... https://lpc.events/event/11/contributions/983/attachments/759/1421/Doing%20more%20with%20lore%20and%20b4.pdf https://lpc.events/event/11/contributions/983/attachments/75...
sarafoster90 4y ago
[flagged]

277 points by taviso 4y ago

https://lkml.org/lkml/2023/2/22/982 https://lkml.org/lkml/2023/2/22/982 Seems to be a known errata which was fixed by a microcode update

sp332 4y ago
https://mobile.twitter.com/taviso/status/1630695259935219713 https://mobile.twitter.com/taviso/status/1630695259935219713 Apparently some Ryzen models have no fixed microcode available. You can boot with clearcpuid=xsaves as a workaround, probably at some performance cost.
- stefan_ 4y ago
  As I understood the email thread, they do have microcode updates but they weren't actually released anywhere but in some crusty vendors BIOS update, so you can only get them if someone fished them out of there. i.e. thats what this repo seems to be: https://github.com/platomav/CPUMicrocodes https://github.com/platomav/CPUMicrocodes
  - deleted 4y ago
    [deleted]
  - caf 4y ago
    The person who reported the bug in their family 0x17 model 0x60 Renoir SoC said it wasn't fixed even by the latest BIOS-supplied microcode available.
    - lathiat 4y ago
      The reality is most consumer motherboards rarely post updates especially after the first year or so. You'll tend to get updates to fix CPU compatability for newer CPUs if the motherboard is still on sale, but otherwise most long term BIOS updates seem largely to be from enterprise vendors (Dell, Lenovo, etc) and much less common on consumer or gaming type hardware. I think most people rely on the operating system to (amazingly) hot patch it during boot. Intel and AMD both publish them and are integrated regularly into most distros (and the linux-firmware git tree). Surprising/weird that they haven't released the Renoir ones. Also seems Tavis had a bug where Debian wasn't applying them on boot for one reason but didn't give details. Wonder what it was.
      - Semaphor 4y ago
        > The reality is most consumer motherboards rarely post updates especially after the first year or so I can’t confirm that. My current board is the MSI X570-A PRO. First BIOS was 2019-06-20, the latest 2022-08-19. And that’s still updating versions and settings. After 3 years, and I’m expecting more. This has also been my experience with other boards. MB updates tend to last several years.
      - kllrnohj 4y ago
        You have that like exactly backwards. You'll get a lot fewer bios updates from Dell or Lenovo than you will from MSI, Asus, Gigabyte, etc.. consumer / gaming motherboard lines. My 5 year old X370-F GAMING is still getting BIOS updates. Others, like MSI, practically forced AMD to continue issuing AGESA updates for X370 & X470 chipsets after AMD had announced official end of support - they got AMD to change course and add new CPU support to those old chipsets. But otherwise all the major consumer / gaming motherboards pick up new AGESA updates quickly & consistently, even when they're EOL platforms.
      - oynqr 4y ago
        I have a really old GA-Z87X-UD5H, it got NVME support five years after release. Probably by accident, but still.
  - jabl 4y ago
    Since this is about Linux, the microcodes it applies on boot can be found in the linux-firmware repository. For AMD microcode, in particular https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/amd-ucode https://git.kernel.org/pub/scm/linux/kernel/git/firmware/lin... However, for some unexplainable reason AMD doesn't tend to update the microcodes in that repo particularly often, leaving it up to BIOS vendors and users updating their BIOS.
- amluto 4y ago
  No obvious reason for a performance cost. If XSAVEC is available, performance should be essentially identical to XSAVES.
- BoardsOfCanada 4y ago
  I had to look up the difference between XSAVES and XSAVEC: "Execution of XSAVES is similar to that of XSAVEC. XSAVES differs from XSAVEC in that it can save state components corresponding to bits set in the IA32_XSS MSR and that it may use the modified optimization."
  - caf 4y ago
    In this case it seems like the "modified optimization" is where the bug lies.
caf 4y ago
It's erratum #1386 (see https://www.amd.com/system/files/TechDocs/56323-PUB_1.00.pdf https://www.amd.com/system/files/TechDocs/56323-PUB_1.00.pdf ): The XSAVES instruction may fail to save XMM registers to the provided state save area if all of the following are true: • All XMM registers were restored to the initialization value by the most recent XRSTORS instruction because the XSTATE_BV[SSE] bit was clear. • The state save area for the XMM registers does not contain the initialization state. • The value in the XMM registers match the initialization value when the XSAVES instruction is executed. • The MXCSR register has been modified to a value different from the initialization value since the most recent XRSTORS instruction.
pabs3 4y ago
Linux folks also plan to apply a workaround for systems running old microcode.
alex_duf 4y ago
As an outsider from the hardware world, I find it astounding that it's possible to fix the behaviour a CPU instruction by changing code. (assuming I understand correctly) In my mind a CPU instruction is hardwired on the chip, and it blows my mind that we keep finding workarounds to already released hardware. Maybe someone could dumb that down for me?
- fhars 4y ago
  The CPU only pretends to be a CPU. In reality, it is a small datacenter comprised of several small special purpose computers doing all the work. I gave up understanding CPUs in depth by the time I read an introducion to Intel's then-new i860 CPU in the April issue of a magazine and it turned out to be a real device.
  - psychphysic 4y ago
    Best explanation in my opinion. Almost like a CPU has an JIT emulator for x86.
    - yvdriess 4y ago
      ucode cracking is relatively straightforward, I wouldn't call it a JIT. https://intelxed.github.io/ref-manual/ https://intelxed.github.io/ref-manual/ It's more like an intepreter for x86, seeing as it is actually executed on a dataflow architecture.
      - psychphysic 4y ago
        Macro op fusion is more what I had in mind when I called it JIT and not interpretation. Not sure what the distinction is but that's where it is to my mind
        yvdriess 4y ago
        It's true that the distinction is a bit vague, the term JIT is overloaded to enough that it stopped being a useful technical term. Compared to 'JVM JIT' or 'LuaJIT': there is no instrumentation to detect what is hot or not. The CPU frontend will crack x86 instructions into micro-ops, while looking for some patterns to merge certain instructions or uops into larger micro-ops. The micro-coded instructions (like many of the legacy instructions) are likely just lookups. Most of this is my speculation, mind. Modern CPU frontends are still kind of a black magic box to me, but I think they are limited by relatively simple transformations by virtue of being on the critical execution path.

snvzz 4y ago

Lovely reminder of the mess x86 is. fp registers shared with mmx, sse registers (xmm), avx registers (ymm), and a truckload of them. Modern implementations have extremely complex frontends, full of elaborate hacks to get performance despite x86. Complexity breeds bugs, such as this one.

userbinator 4y ago
Look at the errata for typical ARM SoCs if you think x86 is bad. A lot of them aren't even publicly available.
reisse 4y ago
I wonder how much performance is really "despite" x86, and how much is thanks to it. To this day, x86 CPUs absolutely dominate everything in compute power. Sure, ARMs are better in performance-per-watt game, but the question of how to scale them to the level of high-end x86 desktop processors is still open. For now, I'd argue it's not even clear if that's possible.
- nine_k 4y ago
  Aren't Apple M2 chips solid desktop-level processors based on ARM?
  - api 4y ago
    High end Intel and AMD chips hold the performance crown but the M chips utterly destroy them on performance per watt. It’s not even close. It’s a mix of a simpler ISA, good core design, and small process nodes.
    - erik 4y ago
      > It’s a mix of a simpler ISA, good core design, and small process nodes. It's also a design that prioritizes perf/watt, whereas CPU vendors tend to prioritize perf/area. (aka perf/$)

naikrovek 4y ago

tavis is everywhere in this space. respect.

deleted 4y ago

[deleted]

userbinator 4y ago

Unless you absolutely need the newest for some reason, I think buying CPUs which have been out for a year if not more would be the best way of avoiding hardware bugs like this, as it seems like they've also started using users for random spot-testing instead of verifying against a spec. One would hope they have plenty of esoteric and freely available x86/PC software to use for regression testing, like demoscene productions going back to the 80s (a great way to exercise opcode sequences that compilers might rarely or never create, but should still work), but with the recent CPU bugs, it feels more like "Windows/Linux/$common_OS with $common_software seems to work, ship it!"

scottlamb 4y ago
I think this is solid advice but hardly a guarantee. Note this is discussing Zen2, not Zen4. wikipedia says [1] Zen2 Launched 7 July 2019; 3 years ago. [1] https://en.wikipedia.org/wiki/Zen_2 https://en.wikipedia.org/wiki/Zen_2
- Tuna-Fish 4y ago
  Yes, and this bug was fixed years ago. The person who stumbled upon it only did so because the correct microcode update was not being loaded. Although the fact that there apparently exist some APUs that have not had the update applied because of shitty OEMs is very concerning.
  - jabl 4y ago
    Also AMD being shitty and not publishing microcode updates to the linux-firmware repo so users would get access to them despite crappy OEM's or users neglecting to do BIOS updates.
suresk 4y ago
Zen 2 architecture came out in 2019 and this particular processor in 2020. I don't think your advice would work here, and there are definite downsides to waiting too many years after a processor has been released to buy it.
4y ago

Deeg9rie9usi 4y ago

Please don't link lkml.org, use lore instead: https://lore.kernel.org/lkml/Y%2FW4x7%2FKFqmDmmR7@thinkstation.cmpxchg8b.net/#r https://lore.kernel.org/lkml/Y%2FW4x7%2FKFqmDmmR7@thinkstati...

hardware2win 4y ago
It is unreadable, wtf?? Who the hell came with such a layout. Is it really this hard to copy industry standard like github?
- oblio 4y ago
  He he. It's just copied off basic mailing list designs from the 90s (or the 80s?). That design is basically what happens when you tell a kernel dev to write a mailing list archiving UI. It's clean because they value simplicity, but they obviously have ~0 non-CLI UI/UX experience.
  - Deeg9rie9usi 4y ago
    Since you are an UI/UX expert, send a patch to https://repo.or.cz/public-inbox.git https://repo.or.cz/public-inbox.git
    - oblio 4y ago
      Sending patches by mail is also bad UI/UX :-p
VMG 4y ago
why?
- Deeg9rie9usi 4y ago
  lkml.org was ditched some time ago by kernel devs. It had often issues, is cropping messages, has no export function, has sometimes even ads on the site. That's why kernel.org folks run now their own reliable archive. See: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=05a5f51ca566 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... https://lpc.events/event/11/contributions/983/attachments/759/1421/Doing%20more%20with%20lore%20and%20b4.pdf https://lpc.events/event/11/contributions/983/attachments/75...

sarafoster90 4y ago

[flagged]