10 ms·
AMD Ryzen Machine Crashes on a Sequence of FMA3 Instructions
- deleted 10y ago[deleted]
- tmd83 10y agoThe security implications of this as commented on the article seems pretty bad. Wonder if any microcode fix can work or it would need a silicon level fix. The second I presume would be very bad? -- Keeping my earlier comment but it seems there's a microcode fix in testing currently. http://forum.hwbot.org/showpost.php?p=480922&postcount=30 http://forum.hwbot.org/showpost.php?p=480922&postcount=30
- zelon88 10y agoIt's happened to AMD before, and it was pretty bad for their brand. http://www.anandtech.com/show/2477/2 http://www.anandtech.com/show/2477/2
- raverbashing 10y agoIntel had serious problems as well, let's not forget: the Pentium fp bug and the F00F bug (though those are more than a decade old at this point)
- msbarnett 10y agoHaswell and early Broadwell had a bug in its transactional memory extensions so serious that the microcode "fix" simply permanently disabled the instructions. CPU errata like this aren't uncommon.
- m45t3r 10y agoWhile those are the most remembered bugs, almost every CPU have some kinda of CPU errata that can lock up the system in really weird and specific ways. For example, my Intel Ivy Bridge CPU running Linux: $ dmesg | grep -i microcode [ 0.000000] microcode: microcode updated early to revision 0x1c, date = 2015-02-26 [ 1.415548] microcode: sig=0x306a9, pf=0x10, revision=0x1c [ 1.415665] microcode: Microcode Update Driver: v2.2. The fact that most users does not hit those bugs is because modern OS already patches those microcode before execution (like the case above). Of course, if AMD can't fixes those bugs without performance regressions (remember the infamous TLB Bug from earlier Phenoms?) it can be pretty bad. However I don't think the majority of users needs to be too cautious about it.
- yuhong 10y agoSome of which are quite serious: http://yuhongbao.blogspot.ca/2015/06/why-your-core-2-processor-appear-to-not.html http://yuhongbao.blogspot.ca/2015/06/why-your-core-2-process...
- deleted 10y ago[deleted]
- rincebrain 10y agoDon't forget the (other) f00f bug in the Intel Quark![1] [1] - https://en.wikipedia.org/wiki/Intel_Quark#Segfault_bug https://en.wikipedia.org/wiki/Intel_Quark#Segfault_bug
- makomk 10y agoYeah, that was kind of unfortunate as it made the Intel Quark incompatible with all normal Linux distros. I don't think it even had any kind of microcode update facility, being effectively a slightly updated 486.
- rincebrain 10y agoEven with a runtime microcode update facility, no BIOS update equivalent would still make it pretty fatal for packaging except as an entirely distinct build target, because otherwise you'd need to do something kinky like always first booting a custom kernel+initrd target built to involve no locks, load the microcode update, then kexec into the real kernel.
- neogodless 10y agoI don't know if this bug affects Phenom II chips, but I've been running a Phenom II for 8 straight years in my primary desktop with nary a hiccup. Rock solid reliable. Guess I haven't been running the "correct conditions" for a crash?
- rincebrain 10y agoYou could also have a BIOS update with the workaround already baked in, or a chip that has the later silicon without the bug... (I also had a Phenom II for many years without any mysterious crashing problems, even though the chip was the correct revision and there was no BIOS erratum workaround enabled.)
- informedperson 10y agoWindows 7 and later distribute and activate microcode updates through Windows Update.
- rincebrain 10y agoIt seems like it's not a microcode update fix, but an MSR killbit flip from the BIOS? [1] [1] - http://support.amd.com/TechDocs/41322_10h_Rev_Gd.pdf http://support.amd.com/TechDocs/41322_10h_Rev_Gd.pdf page 66
- yuhong 10y agoYes, which reduces performance if I remember correctly. It was quite famous. They eventually released a fixed stepping, and Phenom II is not affected.
- informedperson 10y agoTrue, my mistake. (While it doesn't apply for this specific example, it's still the reason why one generally will see microcode-fixed errata go away rather quickly, independently of BIOS updates)
- 10y ago
- pierrebai 10y agoAMD already has a fix, it requires a BIOS update for the motherboard. They have not given details, but some experts think the fix is merely an increase in the power delivered to that part of the chip. Their conclusion stems from teh fact that the FMA3 bug disappeared when the chip was over-clocked because the over-clocking required increasing the voltage to the chip. So the BIOS update probably adjust the voltages up a bit. http://www.fudzilla.com/news/processors/43166-amd-confirms-ryzen-screen-freeze-fix http://www.fudzilla.com/news/processors/43166-amd-confirms-r...
- mickronome 10y agoSince they are pushing how efficiently the chip are regulating power to various parts of the chip to keep power draw to a minimum it sounds like a likely cause. Assuming most of the testing budget is spent on the most common and likely instruction sequence patterns is probably not unrealistic. Still a bit odd that such a short sequence of repeats of the same instruction failed. Which got me thinking. At more than 2000 opcodes for x86-64 and legacy opcodes, testing all 3-sequences with one input is more than 8G sequences, to cover that and (guesstimated average) 64 bits of input per opcode is a staggering 10^29 number of instructions, or 2T CPU*year at 4Gops/s. Not easy to test all that, even if it would be off by a couple of magnitudes in the right direction! Might be that valid 2-sequences is the most that can hope to be even close to exhaustively tested while at the same time covering some significant fraction of the operand space ? The fact that modern CPU's work at all given their complexity in so many different areas looks pretty darn close to magic, even from a pretty close distance.
- deleted 10y ago[deleted]
- flamedoge 10y agoAt least we try, with SMT solvers and such. Surprise surprise, math works!
- informedperson 10y agoA power issue would also plausibly explain why disabling SMT avoids the problem, since disabling SMT powers off a bunch of stuff (a whole thread context does need a bit of power), and, in a multi-threaded scenario, tends to reduce core usage thereby reducing average (! so likely not relevant) power draw as well.
- __david__ 10y agoIf it's not fixable in microcode, then yeah, I'd say it's pretty bad. Makes it useless as a cloud machine, if anyone can cause a DOS and take out every other VM on a host. For a personal machine, it's probably not terrible if you are using linux or something else where you can compile everything yourself. But running binaries built by someone else would be a crapshoot. I wonder how many games (of the Windows, AAA variety) use these multiply instructions? There's also not a lot of detail in the article—like if the data has to be specific or just the combination of instructions is enough to crash it.
- zigzigzag 10y agoCloud doesn't have to mean virtualised and shared. There are plenty of operations that run on dedicated hardware.
- bonzini 10y agoFor many cloud vendors, including EC2, "dedicated hardware" just means yours will be the only virtual machine on that computer. But they're still running your image under a hypervisor.
- zigzigzag 10y agoYes but if you're worried about attacks from co-tenants, that's OK (the cloud provider can always access your hardware anyway)
- bonzini 10y agoIt would still be a pain for cloud providers if the tenant can unwillingly DoS the hypervisor. Sure they have watchdogs in place, but it gets more complicated if the DoS can be non-malicious.
- __david__ 10y agoI suspect Amazon (for instance) has far more shared machines than dedicated machines. If something isn't viable for shared machines then a huge section of the market is ripped away. Still seems catastrophic to me.
- theandrewbailey 10y ago> Flops is only affected when the SMT is enabled, so disabling the SMT can be used as a temporary work-around (until the actual fix arrives). Does this mean disabling SMT will also fix the bug, or is that specific to this app? I've heard that some things improve performance on Ryzen with SMT off, but I've heard that's because OS level task schedulers need to be optimized better. But I still wonder if Ryzen's SMT implementation is on par with Intel's first implementations.
- mhroth 10y agoFMA3 (fused multiply add) instructions are very common for any kind of audio processing (e.g. filters), not only for their speed but also their precision properties. Depending on how a game was compiled, these instructions would be all over the place. But if this specific sequence is every run, that's another question, of course.
- gcp 10y agoFMA is used in literally everything that deals with floats and has AVX2/FMA3 support, not just audio. AVX usage in game code isn't actually all that common, though, due to the need to support lowest common denominators. And no, this specific sequence isn't ever run. If it were, they'd have found it while designing the chip. I checked the original source and it's hand written FMA3 intrinsics that don't correspond to a real computation.
- Animats 10y agoCan you induce this through WebAssembly? Probably not, but someone should check.
- sunfish 10y agoWebAssembly does not currently have an fma instruction, and implementations are not permitted to fold plain multiply+add sequences into fma because that produces different rounding.
- zlynx 10y agoSome game developers that had optimized with AVX were asked to take it out since it would crash overclocked CPUs that were riding the limits. Hit a patch of AVX and bam, they'd fall over. Intel has mostly fixed this in more recent chips so that now they look ahead for upcoming AVX instructions and just slow down.
- geezerjay 10y agoFMA (fused multiply-add) is a basic computing operation that is a basic building block in tasks such as evaluating polynomials, vector-vector and matrix-vector multiplication, convolutions, and also algorithms to solve nonlinear equations. You'd be hard pressed to find any computing application that doesn't take advantage of FMA instructions.
- jjuhl 10y agoWell, it's not like Intel CPUs don't have similar bugs. See Skylake for example - the list of known errata starts on page 27 and continues on through page 63 : http://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/desktop-6th-gen-core-family-spec-update.pdf http://www.intel.com/content/dam/www/public/us/en/documents/...
- yeukhon 10y agoInteresting. I guess unlike software, these sorts of bugs can't be fixed without producing a new one right? Do CPU and GPU manufactures do any types of fuzzing?
- jjuhl 10y agoMicrocode is software running on your CPU. And many problems can be fixed with a microcode update, but not all - see the workarounds listed in the PDF I linked - note especially how many say "None identified".
- yeukhon 10y agoOkay, thanks for clarification. I was thinking about the floating bug which they had to recall the defective processors: https://en.wikipedia.org/wiki/Pentium_FDIV_bug https://en.wikipedia.org/wiki/Pentium_FDIV_bug
- justsid 10y agoHard to say if something like this would be patchable via a microcode update. But regardless, back then your CPU didn’t run software to run your software, so a new hardware revision was in order.
- deleted 10y ago[deleted]
- monocasa 10y agoCPU ones definitely fuzz, and formally verify parts of their chips. Unfortunately fuzzing ultimately has a random component, which doesn't really prove that you got all of these bugs.
- woodson 10y agoIntel's AVX2 instructions don't crash the machine, but lead to an extreme increase in voltage and, thus, temperature. Sustained use of these instructions (say, for example, in signal processing and iterative parameter estimation algorithms implemented using routines in Intel's MKL) requires cooling way beyond what the usual boxed cooler that comes with an i7 CPU can deliver in order to prevent throttling.
- TazeTSchnitzel 10y agoProbably part of why (the other obvious reason being die space) Ryzen lacks 256-bit SIMD hardware, and instead runs such instructions with a 128-bit unit over two cycles.
- informedperson 10y agoWell doing that tends to halve throughput for ALU-dense code, but obviously saves a bit of die space and more importantly reduces peak power draw, which has to be supported by the metallization on the chip. AVX2 might have also been just a bit too late to be integrated outside the microcode.
- TazeTSchnitzel 10y agoI wonder how much worse it is in practice, because Intel's chips supposedly have to downclock a bit when doing AVX2 because of the heat.
- informedperson 10y agoDesktop parts have no dedicated AVX clock (however, their base vs. all-core turbo pretty much does the same thing), while server SKUs (Socket R3 and later) basically have two distinct clocking ladders, one for non-AVX code and one for AVX code (the "mode switch" quantum is rather coarse, on the order of one millisecond). So both base clock and maximum turbo clocks are lower in AVX code compared to non-AVX code. As usual, actual achieved clock speeds depend on thermal performance and are not fixed. If cooling is completely insufficient, the clock frequency will drop below base clocks, i.e. thermal throttling. To clarify: This isn't a secret, but part of the specs and even in some Intel slides. (However, platform and processor specs, even at a basic (product D/S) level, tend to be a rather lengthy read, so most do away with reading all that).
- CalChris 10y agoI think the original hwbot posting is the better article. http://forum.hwbot.org/showthread.php?t=167605 http://forum.hwbot.org/showthread.php?t=167605 The original post has been updated as more facts came along. It verified that the bug was reproducible on other machines. And then it said: Update 3/16/2017: As much as I had least expected this to be the case, this appears to have been confirmed as an errata in the AMD Zen processor. And then it goes on to say: Fortunately, it's one that is fixable with a microcode update and will not result in something catastrophic like a recall or the disabling of features. Basically, this is an awesome bug report. HN should link to it rather than the techpowerup article.
- sriramkarnati 10y agoThe issue seems to be fixed: http://forum.hwbot.org/showpost.php?p=480922&postcount=30 http://forum.hwbot.org/showpost.php?p=480922&postcount=30 http://forum.hwbot.org/showpost.php?p=480524&postcount=25 http://forum.hwbot.org/showpost.php?p=480524&postcount=25 https://www.bit-tech.net/news/hardware/2017/03/21/amd-ryzen-fma3-fix-promise/1 https://www.bit-tech.net/news/hardware/2017/03/21/amd-ryzen-...
- gghh 10y agoah! the author of that hwbot.org post is Alex Yee; I remember his name as he wrote that insanely upvoted (and awesome) stackoverflow answer on branch misprediction: http://stackoverflow.com/a/11227902 http://stackoverflow.com/a/11227902
- tedunangst 10y agoUnfortunately, it's just a full page ad telling me to install some app. back
- i336_ 10y agoCan concur - multiple comments in the linked article show confusion as as to why this is even a problem "because the benchmark program in question has CPU-series-specific binaries, and it didn't release one for Ryzen yet" (paraphrasing). Ouch. Whoosh...
- dang 10y ago
- filereaper 10y agoLike other have mentioned, this happens, to both Intel, AMD and others. Intel had similar failures like its TSX instruction set which was only fixed in the next stepping. Not clear if AMD can push out a patch to the microcode to fix this.
- deleted 10y ago[deleted]
- late2part 10y agoPatient: Dr. It hurts when I do "this." Dr.: Don't do that.
- WhitneyLand 10y agoIs it known if he microcode patch affects performance?
- rurban 10y agoCan someone please at fix the title? Crashes -> Freezes. Big difference. And summarized: Intel has about 80 known similar problems as outlined in the errata, for which no known fixes exist. For this AMD bug a Bios update fix exists (more voltage).
- ysleepy 10y agoSome context relating to Intel: "We saw some really bad Intel CPU bugs in 2015, and we should expect to see more in the future" https://danluu.com/cpu-bugs/ https://danluu.com/cpu-bugs/ https://news.ycombinator.com/item?id=10877270 https://news.ycombinator.com/item?id=10877270