19 ms·

Intel Skylake/Kaby Lake processors: broken hyper-threading

1108 points by vbernat 9y ago

riledhel 9y ago
Does Windows have a patch for this too? Or just disabling HT is the safest option?
- lima 9y ago
  It's safe to assume that Windows distributes the microcode fixes as well.
  - hedora 9y ago
    For some reason, Debian says you need bios updates for certain machines, and Debian packaged microcode for others. I wouldn't be surprised if it is the same for Windows.
- Nexxxeh 9y ago
  By my reading, it's fixed in Micro-code, so if you Windows Update, you already have it fixed.
  - pvdebbe 9y ago
    I wonder if one could update the microcode using a ring 0 virtualization platform and Windows as a guest.
    - my123 9y ago
      Nope, can only be done in the host and not VMs for security reasons.
      - caf 9y ago
        ..and microcode updates aren't persistent, they have to be applied every boot anyway.
- kronos29296 9y ago
  If they don't fix for Windows then they are killing 90% of their customers which they won't. So if it's not already out it will be soon. It is Linux which gets fixes late most of the time.
- pbsd 9y ago
  Windows does have a microcode update driver, as you would expect, so it can fix this. However, looking at the microcode update driver on an updated Windows 10 as of right now, I don't see a recent enough microcode version to fix it. The latest updates appear to be from 2015.
  - sqldba 9y ago
    That's pretty cool, I didn't know of this functionality. Is there a good resource on how to look it up?
    - pbsd 9y ago
      I haven't seen a description of it anywhere that I can think of. The driver lives in C:\Windows\system32\mcupdate_{genuineintel,authenticamd}.dll. All it does is detect which CPU it's running on, and load the appropriate microcode. Loading the microcode is straightforward---all you need to do is put a few values in the appropriate MSRs. This is described in the Intel manuals, Volume 3. The microcode itself is embedded in the above DLLs, as a big binary table. The latest entry I see is 20150812 for the Intel 0x40651, aka Haswell.
boris 9y ago
Intel treats its customers like mushrooms: feeds them shit and keeps them in the dark.
rwmj 9y ago
So if I understand correctly, some affected processors can be fixed by a microcode update, but there are some which cannot be fixed at all? Also the advisory seems to imply that the OCaml compiler uses gcc for code generation, which it does not -- it generates assembly directly, only using gcc as a front end to the linker.
- bhouston 9y ago
  What is the percentage of non fixable chips for Skylake and kabylake? Sometimes those early steppings are not widely distributed.
  - mschuster91 9y ago
    According to the mail, the systems "cannot be fixed" because they lack HyperThreading in the first place so there is no fix to apply.
- dooglius 9y ago
  It sounds like they can all be fixed by disabling hyperthreading.
  - libeclipse 9y ago
    > fixed mitigated
  - a3_nm 9y ago
    Assuming the BIOS/UEFI has an option for it... my laptop doesn't seem to have one (and there are no updates to fix the bug either...).
- brianwawok 9y ago
  Sounds like they can be fixed by the patch isn't out yet?
- tom_mellior 9y ago
  > advisory seems to imply that the OCaml compiler uses gcc for code generation, which it does not -- it generates assembly directly Yes, but that assembly code contains calls into the OCaml runtime, for garbage collection etc. If I understand correctly, the particular loop affected by this bug was somewhere in this memory management code. That code is written in C and compiled with a C compiler.
natehouk 9y ago
So if I understand correctly, this was known all along?
userbinator 9y ago
The problem description is short and scary: Problem: Under complex micro-architectural conditions, short loops of less than 64 instructions that use AH, BH, CH or DH registers as well as their corresponding wider register (e.g. RAX, EAX or AX for AH) may cause unpredictable system behavior. This can only happen when both logical processors on the same physical processor are active. I wonder how many users have experienced intermittent crashes etc. and just nonchalantly attributed it to something else like "buggy software" or even "cosmic ray", when it was actually a defect in the hardware. Or more importantly, how many engineers at Intel, working on these processors, saw this happen a few times and did the same. More interestingly, I would love to read an actual detailed analysis of the problem. Was it a software-like bug in microcode e.g. neglecting some edge-case, or a hardware-level race condition related to marginal timing (that could be worked around by e.g. delaying one operation by a cycle or two)? It reminds me of bugs like https://news.ycombinator.com/item?id=11845770 https://news.ycombinator.com/item?id=11845770 This and the other rather scary post at http://danluu.com/cpu-bugs/ http://danluu.com/cpu-bugs/ suggests to me that CPU manufacturers should do more regression testing, and far more of it. I would recommend demoscene productions, cracktros, and even certain malware, since they tend to exercise the hardware in ways that more "mainstream" software wouldn't come close to. ;-) (To those wondering about ARM and other "simpler" SoCs in embedded systems etc.: They have just as much if not more hardware bugs than PCs. We don't hear about them often, since they are usually worked around in the software which is usually customised exactly for the application and doesn't change much.)
- wbl 9y ago
  CPU manufacturers do do huge amounts of testing, and Intel does formal verification of some functional units. The reliability is far better than most software, in part because making a new release costs billions.
  - wyager 9y ago
    This seems like precisely the sort of thing that a competent manufacturer should rule out formally. Formal verification of individual FUs isn't exactly ambitious... I think we're getting to levels of complexity where the process Intel uses, with lots of different QA and testing teams doing their best to look for bugs, just isn't going to cut it. We need formally verified models transformed step-by-verified-step all the way down to the silicon. It's already feasible, with free tools, to formally verify your high-level model (using e.g. LiquidHaskell) and then transform this to RTL (using e.g. Clash). With Intel's QA/testing budget, it's well within reach to A) verify the transformation steps and B) figure out how to close the performance gap between machine-generated (but maybe slower) and hand-rolled (faster, but evidently wrong) silicon.
isaac_is_goat 9y ago
Holy cow. Definitely feel like I dodged a bullet by building an AMD/Ryzen system this time around - which had it's own set of issues (but seem to be more or less ironed out now).
- brianwawok 9y ago
  I don't think Ryzen has had any more or less bugs than the recent Intel generation. They are at least on the same order of magnitude. Ryzen is great, I might buy one next, but it is not to "doge bullets".
  - floatboth 9y ago
    doge? :D
    - brianwawok 9y ago
      I need a better android keyboard
- gbin 9y ago
  This is not a fair comment: Ryzen had a crash that can be triggered by compiling with GCC and a memory compatibility issue where it cannot run them at their nominal speed. Ryzen is a really young architecture, it already had like 6 stable patches of microcode and you can expect way more.
  - isaac_is_goat 9y ago
    Still seems to be doing better than "turn off half of your processor, sorry no fix just buy a new one".
    - pvdebbe 9y ago
      Hyperthreads are just that - threads. It won't be 50 % slower with HT disabled.
walterbell 9y ago
Does this affect execution of Ocaml runtime, or only the Ocaml compiler?
- foepys 9y ago
  It seems like everything that uses less than 64 instructions in a loop while using the AH, BH, CH or DH registers is affected.
- rwmj 9y ago
  It affects code generated by the OCaml native compiler, which includes the OCaml compiler itself since it is written in OCaml. See also: https://caml.inria.fr/mantis/view.php?id=7452 https://caml.inria.fr/mantis/view.php?id=7452 It also affects code generated by GCC, but apparently GCC is less likely to generate code sequences which trigger the CPU bug.
  - tom_mellior 9y ago
    > It affects code generated by the OCaml native compiler Possibly, but that was not the issue here. It's clear from your own link that the crashes were due to C code in the OCaml runtime (used by the compiler itself), which is written in C and was compiled with GCC at -O2. See https://caml.inria.fr/mantis/view.php?id=7452#c17129 https://caml.inria.fr/mantis/view.php?id=7452#c17129
    - rwmj 9y ago
      Indeed, you're right.
ourcat 9y ago
So will this be affecting most Macbook Pros of the past few years? If so, there's a way to disable hyper-threading, but you need Xcode (Instruments). Open Instruments. Go to Preferences. Choose 'CPU'. Uncheck "Hardware Multi-Threading". Rebooting will reset it.
- rayiner 9y ago
  No. Even though Skylake came out in 2015, Apple used an older CPU (Haswell) until the October 2016 MacBook Pros.
  - reaperhulk 9y ago
    They used Skylake in late 2016 MBPs and Kaby in the 2017 models.
    - isatty 9y ago
      I'm on a Kaby Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz I haven't seen any erratic crashes yet on code compiled by LLVM 8.1.0 on multiple projects (ROS, Qt5). I hope a microcode fix is pushed by Apple on behalf of Intel soon.
      - pedrocr 9y ago
        Are you from the future? Isn't LLVM currently at 4.0 and going for 5.0 soon?
        sclangdon 9y ago
        macOS comes with Apple's build of Clang, which has different versioning from "proper" Clang.
        pedrocr 9y ago
        That's going to be mess when LLVM proper catches up to those version numbers...
salex89 9y ago
Great, pay a premium for the top of the line CPU to get anything more than 4 threads, that disable it...
- Frenchgeek 9y ago
  Only until a fix is available, if it isn't already yet.
  - Godel_unicode 9y ago
    It is, the issue being fixed was the only feedback the OCaml community received.
- coldtea 9y ago
  Premium != perfect. You pay for the extra speed, cache, etc. And since a cheaper cpu can have the same or worse bugs, the point is moot. You don't pay i7 vs i5 etc for better quality control. For that you have to go to enterprise/server grade stuff that costs much more (Xeon). And even there, nothing is guaranteed to be perfect. In fact, with the complexity of modern CPUs/GPUs it's a minor miracle that anything works at all. We have it better than ever...
onli 9y ago
Rule of thumb: On a desktop, if you have an i5 you do not have Hyperthreading. All i3s and i7s do have Hyperthreading, as do new Kaby Lake Pentiums (G4560, 4600, 4620). On laptops, some i5s are not real quad cores but dual cores with Hyperthreading.
- decisiveness 9y ago
  >Rule of thumb: On a desktop, if you have an i5 you do not have Hyperthreading. All i3s and i7s do have Hyperthreading, as do new Kaby Lake Pentiums (G4560, 4600, 4620). Hmm...either this statement is wrong or this desktop /proc/cpinfo is wrong: $ grep -E 'model|stepping|cpu cores' /proc/cpuinfo | sort -u cpu cores : 4 model : 94 model name : Intel(R) Core(TM) i5-6600 CPU @ 3.30GHz stepping : 3 $ grep -q '^flags.*[[:space:]]ht[[:space:]]' /proc/cpuinfo && echo "Hyper-threading is supported" Hyper-threading is supported Intel's product spec page[1] lists this CPU as not supporting Hyper-Threading so I'm a bit puzzled as to why the ht flag is present. [1]https://ark.intel.com/products/88188/Intel-Core-i5-6600-Processor-6M-Cache-up-to-3_90-GHz https://ark.intel.com/products/88188/Intel-Core-i5-6600-Proc...
  - my123 9y ago
    Do you have 8 threads?
  - justinclift 9y ago
    Hmmm, checking for "ht" seems to be giving weird info. On a i5-750 here (few years old), running Fedora 25: $ grep '^flags.*[[:space:]]ht[[:space:]]' /proc/cpuinfo flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm tpr_shadow vnmi flexpriority ept vpid dtherm ida "ht" is being returned even though the CPU only has 4 cores and no hyperthreading: https://ark.intel.com/products/42915/Intel-Core-i5-750-Processor-8M-Cache-2_66-GHz https://ark.intel.com/products/42915/Intel-Core-i5-750-Proce... dmidecode seems to give more accurate info for this: $ sudo dmidecode -t processor | grep 'Count:' Core Count: 4 Thread Count: 4
angry_octet 9y ago
Am I hellbanned?
- deleted 9y ago
  [deleted]
- jacquesm 9y ago
  nope
  - angry_octet 9y ago
    I was for a while there. Heisenban?
    - jacquesm 9y ago
      What makes you think you were?
      - angry_octet 9y ago
        Couldn't see multiple comments I made in unrelated threads when reading from an incognito tab. Unless comment delay for non logged in viewers is an undocumented feature? Or this is inception level hellban, where some bots and devil curators lurk. What is even real. Btw good to see your Lego contraption article in IEEE Spectrum.
        vertex-four 9y ago
        Pages are cached for significant lengths of time for non-logged-in users.
        jacquesm 9y ago
        > Btw good to see your Lego contraption article in IEEE Spectrum. Oh cool, thank you I didn't even realize it had been published. It's already outdated, the latest re-incarnation has double the drop-off bins for twice the sorting speed.
octoploid 9y ago
Well, at least Intel acknowledges, documents and finally fixes these CPU bugs (via microcode updates). AMD on the other hand doesn't even acknowledge an issue when multiple customers report problems. See this Ryzen bug: https://community.amd.com/thread/215773 https://community.amd.com/thread/215773
- justinclift 9y ago
  Huh? There are plenty of AMD employees in that thread who have acknowledged the problem. They're "looking into it", but just seem to have no progress to report yet, and suck at keeping people in the loop.
  - lstamour 9y ago
    And at least one person near the end of the thread reports the problem occurs when ASLR is turned on, for what it's worth: https://community.amd.com/thread/215773?start=105&tstart=0 https://community.amd.com/thread/215773?start=105&tstart=0
- tyingq 9y ago
  >at least Intel acknowledges Not in this case. "Apparently, Intel had indeed found the issue, documented it (see below) and fixed it. There was no direct feedback to the OCaml people, so they only found about it later."
  - ams6110 9y ago
    It was acknowledged, and documented in a errata bulletin, but not communicated back to the reporter. Assuming that it had not already been reported, discovered internally, or reported by another source. Seems more like sloppy followup on a bug report, which unfortunately happens in any large project.
- broknbottle 9y ago
  The first reply is from an AMD employee..........
theGimp 9y ago
The issue was being investigated by the OCaml community since 2017-01-06, with reports of malfunctions going at least as far back as Q2 2016. It was narrowed down to Skylake with hyper-threading, which is a strong indicative of a processor defect. Intel was contacted about it, but did not provide further feedback as far as we know. Fast-forward a few months, and Mark Shinwell noticed the mention of a possible fix for a microcode defect with unknown hit-ratio in the intel-microcode package changelog. He matched it to the issues the OCaml community were observing, verified that the microcode fix indeed solved the OCaml issue, and contacted the Debian maintainer about it. Apparently, Intel had indeed found the issue, *documented it* (see below) and *fixed it*. There was no direct feedback to the OCaml people, so they only found about it later. Inexcusable.
- lorenzhs 9y ago
  They forgot to follow up on a support ticket. As your quotation mentions, the issue was documented and fixed. Calling that "inexcusable" is a bit strong, don't you think? I'm not a particularly big fan of Intel's practices, but the reactions in this thread seem a bit too strong to me.
  - aeleos 9y ago
    It wouldn't be a big deal if this was just software, but the fact that Intel allowed a PROCESSOR bug to be reported, tested, and fixed without telling anyone that the bug actually exists is honestly horrible. You can't just let CPU bugs under the run since it can throw the stability and reliability of the entire system into question. The people that reported it shouldn't have to dig through Intel microcode updates and test different fixes to see if the bug they found was fixed. Hardware manufacturers (especially processor manufacturers) need to be held to a higher standard when it comes to bug reporting and this kind of behavior really has no excuse.
    - lorenzhs 9y ago
      > Intel allowed a PROCESSOR bug to be reported, tested, and fixed without telling anyone that the bug actually exists That's just not true. Intel published the erratum in April: https://www3.intel.com/content/dam/www/public/us/en/documents/specification-updates/desktop-6th-gen-core-family-spec-update.pdf https://www3.intel.com/content/dam/www/public/us/en/document... - search for SKL150. It was also clearly noted in Debian's intel-microcode changelog on May 15: http://metadata.ftp-master.debian.org/changelogs/non-free/i/intel-microcode/intel-microcode_3.20170511.1_changelog http://metadata.ftp-master.debian.org/changelogs/non-free/i/... They absolutely should have followed up to the OCaml people's support ticket. But sloppy followup is an issue that every large project encounters.
fotcorn 9y ago
The latest intel-microcode package from Ubuntu 16.04 does not fix the problem. I installed the same package from Ubuntu 17.10 [0] which fixes the problem. You can check your system with the script linked in the mailing list thread [1]. [0] https://packages.ubuntu.com/en/artful/amd64/intel-microcode/download https://packages.ubuntu.com/en/artful/amd64/intel-microcode/... [1] https://lists.debian.org/debian-devel/2017/06/msg00309.html https://lists.debian.org/debian-devel/2017/06/msg00309.html
- edwinksl 9y ago
  Launchpad bug report for this: https://bugs.launchpad.net/ubuntu/+source/intel-microcode/+bug/1700373 https://bugs.launchpad.net/ubuntu/+source/intel-microcode/+b...
- arde 9y ago
  Indeed, the latest Intel microcode published for Ubuntu 16.04 is the ancient 20151106 [1]. Later Ubuntu releases do have more recent microcode packages [2]. I cannot understand why they left out 16.04 there. So much for LTS, it seems. This recently came to my attention while debugging some increasingly frequent lockups, which took me a solid week of eliminating all seemingly more likely causes (VirtualBox, nVidia driver, faulty RAM, etc). In the end I found the culprit while digging into the Intel Specification updates: my Core i7-5820K (and most other Haswell-E and Broadwell processors) has a bug when leaving package C-states, and the only workaround is to disable C-states above level 1. Timely updated microcode, which applies this workaround, would have saved me my week. [1] https://launchpad.net/ubuntu/xenial/+source/intel-microcode https://launchpad.net/ubuntu/xenial/+source/intel-microcode [2] https://launchpad.net/ubuntu/+source/intel-microcode/+changelog https://launchpad.net/ubuntu/+source/intel-microcode/+change...
  - rlpb 9y ago
    > Indeed, the latest Intel microcode published for Ubuntu 16.04 is the ancient 20151106. By ancient, perhaps you mean the version that was current at the time 16.04 shipped? > I cannot understand why they left out 16.04 there. So much for LTS, it seems. See https://wiki.ubuntu.com/StableReleaseUpdates https://wiki.ubuntu.com/StableReleaseUpdates. The point of an LTS (or any stable release, for that matter) is that it doesn't change by default. For those who want to keep everything up-to-date, Ubuntu ships a new release on a six month cadence. If you choose not to use that, then you shouldn't be surprised when things aren't updated, since that's exactly what you opted in to. The microcode package may warrant an exception, however, and we have a bug to track that. It's tricky because without the source we cannot pick apart what changed, or determine whether any changes meet our update policy. We have to be careful. Sooner or later some user will inevitably come along to tell us that a microcode update broke things, and ask why we didn't fulfill our LTS promise by not changing it.
tyingq 9y ago
There's a perl script on the debian mailing list that digs a bit deeper and tells you if you're affected in the first place, if you're affected but patched already, affected but have HT disabled, etc. https://lists.debian.org/debian-user/2017/06/msg01011.html https://lists.debian.org/debian-user/2017/06/msg01011.html
- fattire 9y ago
  I ported this to bash since I have a chromebook w/o perl and (as for right now) the fs is read-only, so I just piped the script to it and sure enough my brand-new Samsung Chromebook Pro appears to be vulnerable, though apparently patchable. Details and if you want the script I link to it from here: https://forum.xda-developers.com/hardware-hacking/chromebooks/samsung-chromebook-pro-asus-c302ca-t3627253 https://forum.xda-developers.com/hardware-hacking/chromebook... - don't judge my shitty bash skills. ft
Syzygies 9y ago
In my experience with parallel code written in Haskell, hyper-threading offers only a very mild speedup, perhaps 10%. It is essentially an illusion, a logical convenience. (How long does it take to complete a parallel task on a dedicated machine? Four cores with hyper-threading off has nearly the performance of eight virtual cores with hyper-threading on.) Many people have neither the interest nor the hardware access to overclock, and these processors have less overclocking headroom than earlier designs. Nevertheless, the hyper-threading hardware itself generates heat, restricting the overclocking range for given cpu cooling hardware. In this case, turning off hyper-threading pays for itself, because one can then overclock further, overtaking any advantage to hyper-threading.
- barrkel 9y ago
  It depends on what resources your code uses on-chip. If all threads are contending on the same resources, then you won't see a speedup; if they're using different resources, hyperthreading can increase throughput significantly. I've seen hyperthreading give me the equivalent of 50% of another CPU, particularly when I'm running multiple CPU-bound processes concurrently (so they're not executing the same code at the same time in some kind of parallel operation, and certainly aren't bound on synchronization primitive overheads).
  - Syzygies 9y ago
    That makes sense. I'm a mathematician, and my experience is with pure computations, homogeneous across each (virtual) core.
- esaym 9y ago
  It normally helps "some", and rarely hurts performance. So you might as well enable it.
  - ploxiln 9y ago
    I worry about hyperthreading hurting worst-case latency (since a thread might be assigned to run on a virtual core which does not work as fast as expected).
ncrmro 9y ago
Just got the 2017 no touchbar 13 macbook pro with the kaby lake i7. Should I be worried, can I even disable HT with mac. And presumably the update will be provided so the whole laptop is still ok? I've been using the thunderbolt 3 dock with two external monitors and occasionally get a little glitch prolly loose cable I think. I've downloaded the bitcoin blockchain, done quite a bit of work in pycharm + chrome, multiple projects, flow and webpack in the background and haven't had any sort of crashes tho.
pedrocr 9y ago
Here's how to fix it on a Thinkpad on Linux. I've got a T460s and checked with the script[1] that it was indeed affected. The Debian instructions said to update your BIOS before updating the microcode package so I went to the model support page[2] to the BIOS/UEFI section and downloaded the "BIOS Update (Bootable CD)" one. The changelog included microcode updates so it looked promising[3]. To get the ISO onto a usb drive I did the following: $ geteltorito n1cur14w.iso > eltorito-bios.iso # provided by the genisoimage package on Ubuntu $ sudo dd if=eltorito-bios.iso of=/dev/sdXXX # replace with your usb drive with care to not write over your disk I then had a bootable USB drive that I ran by rebooting the computer, pressing Enter and then F12 to get to the boot drive selection and selecting the USB. From then it's just following the options it gives you. It's basically pressing 2 to go into the update and then pressing Y and Enter a few times to tell it you really want to do it. After that just let it reboot a few times and the update is done. After booting again the same test script[1] now said I had an affected CPU but new enough microcode. [1] https://lists.debian.org/debian-user/2017/06/msg01011.html https://lists.debian.org/debian-user/2017/06/msg01011.html [2] http://pcsupport.lenovo.com/pt/en/products/laptops-and-netbooks/thinkpad-t-series-laptops/thinkpad-t460s/downloads http://pcsupport.lenovo.com/pt/en/products/laptops-and-netbo... [3] https://download.lenovo.com/pccbbs/mobiles/n1cur14w.txt https://download.lenovo.com/pccbbs/mobiles/n1cur14w.txt
geogriffin 9y ago
has anyone affected by this bug tried using a kernel configured with hyperthreading support disabled? would that work?
mjw1007 9y ago
It's painful to have to read text like « select Intel Pentium processor models ». If Intel used marketing names that were more closely related to technical reality, then when something like this happens they wouldn't have so many customers finding themselves in the "maybe I'm affected by this horrid bug" box.
asow92 9y ago
So what does this mean for the thousands of new MacBook Pro 2016/2017 owners out there?
- coldtea 9y ago
  Nothing, since they've been running their laptops without issues (it would have been all over the news if it was some widespread issue) for 2+ years. At some point in the near future Apple will package the microcode fix in an update, and that will be it.
  - thoughtsimple 9y ago
    Probably already done since this was fixed in April/May by Intel. There has been at least one update since then.
  - donkeyd 9y ago
    I've run into an issue where switching users causes a crash on my 2016 MBP. Many more people are having this issue, according to the Google. Also, up until the end of 2016, Apple didn't use Skylake processors, so it wouldn't be 2+ years.
zzalpha 9y ago
Charming. I picked up a 5th Gen X1 Carbon configured with a Kaby Lake processor, and apparently there's no way to disable hyperthreading in the BIOS, and according to Intel's errata, no fix available yet. Oh well... so far the machine (running Windows 10) has been stable minus one or two random lockups in 2 months of heavy usage which could be attributed to this. Guess I wait...
elnik 9y ago
My CPU (6th generation i5) died last week. RIP. I installed debian 9, installed virtualbox, vagrant, setup a clean development machine for myself, everything took 4 hours to finish. I reboot the virtual machine, and boom, there was a kernel panic which I sadly don't remember exactly / didn't take a picture of. After I rebooted the machine, and opened terminal, the system froze. The cursor wouldn't move. Reboot again, motherboard has a CPU fail/undetected light on. Couldn't get it to boot after that. I am both sad and relieved that bad stuff exists, but it's being patched to prevent proliferating. I sincerely hope I'll get a replacement from Intel.
- broknbottle 9y ago
  CPUs rarely die, unless you're OCing or PSU went bad and took things out, I am willing to bet your MoBo is the part that is bad.
luckydude 9y ago
That's a really nicely done announcement. Simple, to the point, no drama, all the info you could want, scripts to figure out your processor, etc. Well done Debian folks!
ComputerGuru 9y ago
So, serious question: If the microcode "fix" for this ends up disabling HT, how does one get a refund not just for the CPU but for the $3k laptop I spec'd around it? Without needing to sue? This isn't a hypothetical; what did Intel do when the only fix for broken functionality was to disable TSX entirely?
- mixmastamyk 9y ago
  I remember the pentium bug in the mid 90s, they actually shipped out replacement processors. Doubt that could be pulled off on laptops. Perhaps a microcode update can work around it.
  - senectus1 9y ago
    Given my Surface Book got a 1/10 repair-ability score on ifixit... I dont think they'll just replace the chip :-P
- paulddraper 9y ago
  The same for any part of your computer, I imagine. What happens when your laptop's display has frequently broken pixels?
  - sqldba 9y ago
    > What happens when your laptop's display has frequently broken pixels? Well they wait for 6 or more before they say it's out of spec
bleair 9y ago
When intel had the floating point division hardware bug they recalled chips. https://en.wikipedia.org/wiki/Pentium_FDIV_bug https://en.wikipedia.org/wiki/Pentium_FDIV_bug I wonder if intel will do something like that again or if the industry as a whole is more tolerant of unreliable / buggy behavior and will just live with it. Examples of Apple just telling people that the poor reception strength was their own fault / changing software to hide problems / etc.
- aarongolliver 9y ago
  If you read the link you'll see they've fixed it with a microcode update.
itsoggy 9y ago
When HT first started appearing on P4 chips I was looking after NetWare, 2K and XP boxes, they would freak out with HT enabled all kinds of oddities, I suspect most because of the OS's not fully supporting it. To this day I disable it by reflex on everything!
cJ0th 9y ago
This is just great! Just yesterday I got a new laptop with a skylake processor. Now I wonder whether I experienced that bug today as I got kicked out of dosbox (on debian) for no apparent reason. In the config file I had to change the value of 'core' in the cpu-section from 'automatic' to 'normal'. Could be something entirely different but it is a funny timing.
whitebowen 9y ago
Do you need a professional hacker, contact cipherhacker1@GMAIL.COM Our service includes Change University grades Facebook, twitter, IG hack Email hack icloud and phone hack * grades change Wipe criminal records Wipe credit card debt MasterCard's/visa cards Bank account *Data base hack and lot more hacking services in general Among other customized services...all this are at all great rate. Results guaranteed. Contact us at cipherhacker1@GMAIL.COM
whitebowen 9y ago
Do you need a professional hacker, contact cipherhacker1@GMAIL.COM Our service includes Change University grades Facebook, twitter, IG hack Email hack icloud and phone hack * grades change Wipe criminal records Wipe credit card debt MasterCard's/visa cards Bank account *Data base hack and lot more hacking services in general Among other customized services...all this are at all great rate. Results guaranteed. Contact us at cipherhacker1 GMAIL.COM
joshschreuder 9y ago
How does one get new microcode on Windows? Is that what the Intel Chipset drivers are? And is the microcode fix available for non-Linux systems yet?
- aarongolliver 9y ago
  Windows update
  - tempestn 9y ago
    Has this already been patched via Windows Update? I didn't even know WU could do CPU microcode patches.
whitebowen 9y ago
Do you need a professional hacker, contact cipherhacker1@GMAIL.COM Our service includes Change University grades Facebook, twitter, IG hack Email hack icloud and phone hack * grades change Wipe criminal records Wipe credit card debt MasterCard's/visa cards Bank account *Data base hack and lot more hacking services in general Among other customized services...all this are at all great rate. Results guaranteed. Contact us at cipherhacker1@GMAIL.COM
age_bronze 9y ago
I would've expected at least an example assembly code reproducing the bug? How was it not discovered before, but only with the OCaml compiler? They say "unexpected behavior", does this mean that code compiled with this can give incorrect results? Can this have any security implication? How much code was compiled with similar patterns? Can the problem reproduced with any JIT compiler? We need to know what can cause this, maybe compiled and working code already contains such patterns waiting to be abused...
wfunction 9y ago
A little off-topic, but does anybody know of any hacky ways to disable hyper-threading (on Haswell if it matters) if the firmware doesn't provide the option?
spektom 9y ago
I wonder how this issue affects cloud providers?
Traubenfuchs 9y ago
Are those microcode updates deployed via windows (10) update?
paines 9y ago
I have a Skylake mobile CPU (i7-6700hq) and it pretty much rocks with Ubuntu 17. Also the system is stable and fast. Under heavy load, e.g. games the system is stable. Compiling a big(>10000 modules) C++ project via ninja/cmake under Qt Creator hangs the system resporducibly after ~15 minutes. I wonder now if this broken hyperthreading could be such sideffect.
herpderperator 9y ago
If anyone on Windows wishes to update their CPU microcode without waiting for Microsoft to push it out via Windows Update, you can use this tool from VMware https://labs.vmware.com/flings/vmware-cpu-microcode-update-driver https://labs.vmware.com/flings/vmware-cpu-microcode-update-d... which can update microcode as well. Windows stores its microcode in C:\Windows\System32\mcupdate_GenuineIntel.dll which is a proprietary binary file and you can't simply replace it with Intel's microcode.dat file (which is ASCII text), so you have to use a third-party tool such as VMware's one. Simply: 1. Download and extract the zip file in the first paragraph 2. Modify the install.bat file so that the line which reads `for %%i IN (microcode.dat microcode_amd.bin microcode_amd_fam15h.bin) DO (` only contains the microcode.dat parameter (since you obviously don't have an AMD CPU, and the tool is made for both) 3. Download and extract microcode.dat from Intel's website (https://downloadcenter.intel.com/download/26798/Linux-Processor-Microcode-Data-File https://downloadcenter.intel.com/download/26798/Linux-Proces...) and place it into the same directory as the VMware tool 4. Run install.bat with admin privileges 5. Hit cancel when it tells you that the AMD microcode files are missing, and you're done The CPU microcode will be updated immediately (yes, while Windows is running.) The service will also run on each boot and update your CPU microcode, since microcode updates are only temporary and are lost each time you restart. You can check Event Viewer for entries from `cpumcupdate` to see what it has done. It's advised to run a tool to view the microcode version before installing (such as HWiINFO64) so you can re-run the tool after installing and confirming that the version has changed. I have done this and it works as described. I went from 0x74 to 0xba as shown by the μCU field in HWiNFO64, and I have an i7-6700k.
wscott 9y ago
Has anyone benchmarked one of these machines before and after applying this microcode update? The options in microcode are rather limited and all are likely to have performance impacts. This is likely disabling functionally to avoid this case. I would hope the patch is smart enough to not apply if threading is not enabled, but who knows.
- jakeogh 9y ago
  Would a performance hit go unnoticed? Usually (always?) it's not a ROM update, the encrypted microcode blob is loaded into the CPU by the OS on every boot via CONFIG_MICROCODE. some linkrot: http://imgur.com/a/z1uLv http://imgur.com/a/z1uLv
jwildeboer 9y ago
I do wonder though, why didn't Debian maintainers pick up the microcode updates when they were made available by Intel? Why did it need a wink from the Ocaml people for them to note? Or am I missing something?
msimpson 9y ago
The late 2016 Razer Blade uses the i7-6700HQ which is specifically a Family 6, Model 94, Stepping 3 processor. I wonder if a microcode update would solve some of the various issues I have in Windows.
karussell 9y ago
What is the probability for this to happen? Or how could I estimate the time it takes for random code to hit this bug at least once with a probability of over 90%?
ManyEthers 9y ago
Intel's communication is incredibly poor. Errata exist for all CPUs but this one is quite important and resulted in no proper public communication it seems.
Magnificents 9y ago
The poor guys from OCaml who found the bug. Imagine how much debugging it takes to find such an issue and narrow it down to the precise register sequence. I guess since it’s a hyper threading bug it even depends on multiple threads doing certain things at the same time. Usually you trust your CPU to execute code properly.
convefefe 9y ago
One of many typical erratum... nothing to see here, been patched months ago. Most people are unlikely to ever encounter it even if unpatched.
peter_retief 9y ago
If I run $grep name /proc/cpuinfo | sort -u model name : Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz then $cat /proc/cpuinfo | grep ht Definitely there under flags Should I be concerned?
- itchyouch 9y ago
  Your processor isn't affected. It would be i[357]-[67]xxx processors only. But a new computer may serve you well...
ericfrederich 9y ago
Is it fixed or not? Beginning of post says to disable hyperthreading but then goes on to say Intel fixed this with a microcode update.

1108 points by vbernat 9y ago

riledhel 9y ago

Does Windows have a patch for this too? Or just disabling HT is the safest option?

lima 9y ago
It's safe to assume that Windows distributes the microcode fixes as well.
- hedora 9y ago
  For some reason, Debian says you need bios updates for certain machines, and Debian packaged microcode for others. I wouldn't be surprised if it is the same for Windows.
Nexxxeh 9y ago
By my reading, it's fixed in Micro-code, so if you Windows Update, you already have it fixed.
- pvdebbe 9y ago
  I wonder if one could update the microcode using a ring 0 virtualization platform and Windows as a guest.
  - my123 9y ago
    Nope, can only be done in the host and not VMs for security reasons.
    - caf 9y ago
      ..and microcode updates aren't persistent, they have to be applied every boot anyway.
kronos29296 9y ago
If they don't fix for Windows then they are killing 90% of their customers which they won't. So if it's not already out it will be soon. It is Linux which gets fixes late most of the time.
pbsd 9y ago
Windows does have a microcode update driver, as you would expect, so it can fix this. However, looking at the microcode update driver on an updated Windows 10 as of right now, I don't see a recent enough microcode version to fix it. The latest updates appear to be from 2015.
- sqldba 9y ago
  That's pretty cool, I didn't know of this functionality. Is there a good resource on how to look it up?
  - pbsd 9y ago
    I haven't seen a description of it anywhere that I can think of. The driver lives in C:\Windows\system32\mcupdate_{genuineintel,authenticamd}.dll. All it does is detect which CPU it's running on, and load the appropriate microcode. Loading the microcode is straightforward---all you need to do is put a few values in the appropriate MSRs. This is described in the Intel manuals, Volume 3. The microcode itself is embedded in the above DLLs, as a big binary table. The latest entry I see is 20150812 for the Intel 0x40651, aka Haswell.

boris 9y ago

Intel treats its customers like mushrooms: feeds them shit and keeps them in the dark.

rwmj 9y ago

So if I understand correctly, some affected processors can be fixed by a microcode update, but there are some which cannot be fixed at all? Also the advisory seems to imply that the OCaml compiler uses gcc for code generation, which it does not -- it generates assembly directly, only using gcc as a front end to the linker.

bhouston 9y ago
What is the percentage of non fixable chips for Skylake and kabylake? Sometimes those early steppings are not widely distributed.
- mschuster91 9y ago
  According to the mail, the systems "cannot be fixed" because they lack HyperThreading in the first place so there is no fix to apply.
dooglius 9y ago
It sounds like they can all be fixed by disabling hyperthreading.
- libeclipse 9y ago
  > fixed mitigated
- a3_nm 9y ago
  Assuming the BIOS/UEFI has an option for it... my laptop doesn't seem to have one (and there are no updates to fix the bug either...).
brianwawok 9y ago
Sounds like they can be fixed by the patch isn't out yet?
tom_mellior 9y ago
> advisory seems to imply that the OCaml compiler uses gcc for code generation, which it does not -- it generates assembly directly Yes, but that assembly code contains calls into the OCaml runtime, for garbage collection etc. If I understand correctly, the particular loop affected by this bug was somewhere in this memory management code. That code is written in C and compiled with a C compiler.

natehouk 9y ago

So if I understand correctly, this was known all along?

userbinator 9y ago

The problem description is short and scary: Problem: Under complex micro-architectural conditions, short loops of less than 64 instructions that use AH, BH, CH or DH registers as well as their corresponding wider register (e.g. RAX, EAX or AX for AH) may cause unpredictable system behavior. This can only happen when both logical processors on the same physical processor are active. I wonder how many users have experienced intermittent crashes etc. and just nonchalantly attributed it to something else like "buggy software" or even "cosmic ray", when it was actually a defect in the hardware. Or more importantly, how many engineers at Intel, working on these processors, saw this happen a few times and did the same. More interestingly, I would love to read an actual detailed analysis of the problem. Was it a software-like bug in microcode e.g. neglecting some edge-case, or a hardware-level race condition related to marginal timing (that could be worked around by e.g. delaying one operation by a cycle or two)? It reminds me of bugs like https://news.ycombinator.com/item?id=11845770 https://news.ycombinator.com/item?id=11845770 This and the other rather scary post at http://danluu.com/cpu-bugs/ http://danluu.com/cpu-bugs/ suggests to me that CPU manufacturers should do more regression testing, and far more of it. I would recommend demoscene productions, cracktros, and even certain malware, since they tend to exercise the hardware in ways that more "mainstream" software wouldn't come close to. ;-) (To those wondering about ARM and other "simpler" SoCs in embedded systems etc.: They have just as much if not more hardware bugs than PCs. We don't hear about them often, since they are usually worked around in the software which is usually customised exactly for the application and doesn't change much.)

wbl 9y ago
CPU manufacturers do do huge amounts of testing, and Intel does formal verification of some functional units. The reliability is far better than most software, in part because making a new release costs billions.
- wyager 9y ago
  This seems like precisely the sort of thing that a competent manufacturer should rule out formally. Formal verification of individual FUs isn't exactly ambitious... I think we're getting to levels of complexity where the process Intel uses, with lots of different QA and testing teams doing their best to look for bugs, just isn't going to cut it. We need formally verified models transformed step-by-verified-step all the way down to the silicon. It's already feasible, with free tools, to formally verify your high-level model (using e.g. LiquidHaskell) and then transform this to RTL (using e.g. Clash). With Intel's QA/testing budget, it's well within reach to A) verify the transformation steps and B) figure out how to close the performance gap between machine-generated (but maybe slower) and hand-rolled (faster, but evidently wrong) silicon.

isaac_is_goat 9y ago

Holy cow. Definitely feel like I dodged a bullet by building an AMD/Ryzen system this time around - which had it's own set of issues (but seem to be more or less ironed out now).

brianwawok 9y ago
I don't think Ryzen has had any more or less bugs than the recent Intel generation. They are at least on the same order of magnitude. Ryzen is great, I might buy one next, but it is not to "doge bullets".
- floatboth 9y ago
  doge? :D
  - brianwawok 9y ago
    I need a better android keyboard
gbin 9y ago
This is not a fair comment: Ryzen had a crash that can be triggered by compiling with GCC and a memory compatibility issue where it cannot run them at their nominal speed. Ryzen is a really young architecture, it already had like 6 stable patches of microcode and you can expect way more.
- isaac_is_goat 9y ago
  Still seems to be doing better than "turn off half of your processor, sorry no fix just buy a new one".
  - pvdebbe 9y ago
    Hyperthreads are just that - threads. It won't be 50 % slower with HT disabled.

walterbell 9y ago

Does this affect execution of Ocaml runtime, or only the Ocaml compiler?

foepys 9y ago
It seems like everything that uses less than 64 instructions in a loop while using the AH, BH, CH or DH registers is affected.
rwmj 9y ago
It affects code generated by the OCaml native compiler, which includes the OCaml compiler itself since it is written in OCaml. See also: https://caml.inria.fr/mantis/view.php?id=7452 https://caml.inria.fr/mantis/view.php?id=7452 It also affects code generated by GCC, but apparently GCC is less likely to generate code sequences which trigger the CPU bug.
- tom_mellior 9y ago
  > It affects code generated by the OCaml native compiler Possibly, but that was not the issue here. It's clear from your own link that the crashes were due to C code in the OCaml runtime (used by the compiler itself), which is written in C and was compiled with GCC at -O2. See https://caml.inria.fr/mantis/view.php?id=7452#c17129 https://caml.inria.fr/mantis/view.php?id=7452#c17129
  - rwmj 9y ago
    Indeed, you're right.

ourcat 9y ago

So will this be affecting most Macbook Pros of the past few years? If so, there's a way to disable hyper-threading, but you need Xcode (Instruments). Open Instruments. Go to Preferences. Choose 'CPU'. Uncheck "Hardware Multi-Threading". Rebooting will reset it.

rayiner 9y ago
No. Even though Skylake came out in 2015, Apple used an older CPU (Haswell) until the October 2016 MacBook Pros.
- reaperhulk 9y ago
  They used Skylake in late 2016 MBPs and Kaby in the 2017 models.
  - isatty 9y ago
    I'm on a Kaby Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz I haven't seen any erratic crashes yet on code compiled by LLVM 8.1.0 on multiple projects (ROS, Qt5). I hope a microcode fix is pushed by Apple on behalf of Intel soon.
    - pedrocr 9y ago
      Are you from the future? Isn't LLVM currently at 4.0 and going for 5.0 soon?
      - sclangdon 9y ago
        macOS comes with Apple's build of Clang, which has different versioning from "proper" Clang.
        pedrocr 9y ago
        That's going to be mess when LLVM proper catches up to those version numbers...

salex89 9y ago

Great, pay a premium for the top of the line CPU to get anything more than 4 threads, that disable it...

Frenchgeek 9y ago
Only until a fix is available, if it isn't already yet.
- Godel_unicode 9y ago
  It is, the issue being fixed was the only feedback the OCaml community received.
coldtea 9y ago
Premium != perfect. You pay for the extra speed, cache, etc. And since a cheaper cpu can have the same or worse bugs, the point is moot. You don't pay i7 vs i5 etc for better quality control. For that you have to go to enterprise/server grade stuff that costs much more (Xeon). And even there, nothing is guaranteed to be perfect. In fact, with the complexity of modern CPUs/GPUs it's a minor miracle that anything works at all. We have it better than ever...

onli 9y ago

Rule of thumb: On a desktop, if you have an i5 you do not have Hyperthreading. All i3s and i7s do have Hyperthreading, as do new Kaby Lake Pentiums (G4560, 4600, 4620). On laptops, some i5s are not real quad cores but dual cores with Hyperthreading.

decisiveness 9y ago
>Rule of thumb: On a desktop, if you have an i5 you do not have Hyperthreading. All i3s and i7s do have Hyperthreading, as do new Kaby Lake Pentiums (G4560, 4600, 4620). Hmm...either this statement is wrong or this desktop /proc/cpinfo is wrong: $ grep -E 'model|stepping|cpu cores' /proc/cpuinfo | sort -u cpu cores : 4 model : 94 model name : Intel(R) Core(TM) i5-6600 CPU @ 3.30GHz stepping : 3 $ grep -q '^flags.*[[:space:]]ht[[:space:]]' /proc/cpuinfo && echo "Hyper-threading is supported" Hyper-threading is supported Intel's product spec page[1] lists this CPU as not supporting Hyper-Threading so I'm a bit puzzled as to why the ht flag is present. [1]https://ark.intel.com/products/88188/Intel-Core-i5-6600-Processor-6M-Cache-up-to-3_90-GHz https://ark.intel.com/products/88188/Intel-Core-i5-6600-Proc...
- my123 9y ago
  Do you have 8 threads?
- justinclift 9y ago
  Hmmm, checking for "ht" seems to be giving weird info. On a i5-750 here (few years old), running Fedora 25: $ grep '^flags.*[[:space:]]ht[[:space:]]' /proc/cpuinfo flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm tpr_shadow vnmi flexpriority ept vpid dtherm ida "ht" is being returned even though the CPU only has 4 cores and no hyperthreading: https://ark.intel.com/products/42915/Intel-Core-i5-750-Processor-8M-Cache-2_66-GHz https://ark.intel.com/products/42915/Intel-Core-i5-750-Proce... dmidecode seems to give more accurate info for this: $ sudo dmidecode -t processor | grep 'Count:' Core Count: 4 Thread Count: 4

angry_octet 9y ago

Am I hellbanned?

deleted 9y ago
[deleted]
jacquesm 9y ago
nope
- angry_octet 9y ago
  I was for a while there. Heisenban?
  - jacquesm 9y ago
    What makes you think you were?
    - angry_octet 9y ago
      Couldn't see multiple comments I made in unrelated threads when reading from an incognito tab. Unless comment delay for non logged in viewers is an undocumented feature? Or this is inception level hellban, where some bots and devil curators lurk. What is even real. Btw good to see your Lego contraption article in IEEE Spectrum.
      - vertex-four 9y ago
        Pages are cached for significant lengths of time for non-logged-in users.
      - jacquesm 9y ago
        > Btw good to see your Lego contraption article in IEEE Spectrum. Oh cool, thank you I didn't even realize it had been published. It's already outdated, the latest re-incarnation has double the drop-off bins for twice the sorting speed.

octoploid 9y ago

Well, at least Intel acknowledges, documents and finally fixes these CPU bugs (via microcode updates). AMD on the other hand doesn't even acknowledge an issue when multiple customers report problems. See this Ryzen bug: https://community.amd.com/thread/215773 https://community.amd.com/thread/215773

justinclift 9y ago
Huh? There are plenty of AMD employees in that thread who have acknowledged the problem. They're "looking into it", but just seem to have no progress to report yet, and suck at keeping people in the loop.
- lstamour 9y ago
  And at least one person near the end of the thread reports the problem occurs when ASLR is turned on, for what it's worth: https://community.amd.com/thread/215773?start=105&tstart=0 https://community.amd.com/thread/215773?start=105&tstart=0
tyingq 9y ago
>at least Intel acknowledges Not in this case. "Apparently, Intel had indeed found the issue, documented it (see below) and fixed it. There was no direct feedback to the OCaml people, so they only found about it later."
- ams6110 9y ago
  It was acknowledged, and documented in a errata bulletin, but not communicated back to the reporter. Assuming that it had not already been reported, discovered internally, or reported by another source. Seems more like sloppy followup on a bug report, which unfortunately happens in any large project.
broknbottle 9y ago
The first reply is from an AMD employee..........

theGimp 9y ago

The issue was being investigated by the OCaml community since 2017-01-06, with reports of malfunctions going at least as far back as Q2 2016. It was narrowed down to Skylake with hyper-threading, which is a strong indicative of a processor defect. Intel was contacted about it, but did not provide further feedback as far as we know. Fast-forward a few months, and Mark Shinwell noticed the mention of a possible fix for a microcode defect with unknown hit-ratio in the intel-microcode package changelog. He matched it to the issues the OCaml community were observing, verified that the microcode fix indeed solved the OCaml issue, and contacted the Debian maintainer about it. Apparently, Intel had indeed found the issue, *documented it* (see below) and *fixed it*. There was no direct feedback to the OCaml people, so they only found about it later. Inexcusable.

lorenzhs 9y ago
They forgot to follow up on a support ticket. As your quotation mentions, the issue was documented and fixed. Calling that "inexcusable" is a bit strong, don't you think? I'm not a particularly big fan of Intel's practices, but the reactions in this thread seem a bit too strong to me.
- aeleos 9y ago
  It wouldn't be a big deal if this was just software, but the fact that Intel allowed a PROCESSOR bug to be reported, tested, and fixed without telling anyone that the bug actually exists is honestly horrible. You can't just let CPU bugs under the run since it can throw the stability and reliability of the entire system into question. The people that reported it shouldn't have to dig through Intel microcode updates and test different fixes to see if the bug they found was fixed. Hardware manufacturers (especially processor manufacturers) need to be held to a higher standard when it comes to bug reporting and this kind of behavior really has no excuse.
  - lorenzhs 9y ago
    > Intel allowed a PROCESSOR bug to be reported, tested, and fixed without telling anyone that the bug actually exists That's just not true. Intel published the erratum in April: https://www3.intel.com/content/dam/www/public/us/en/documents/specification-updates/desktop-6th-gen-core-family-spec-update.pdf https://www3.intel.com/content/dam/www/public/us/en/document... - search for SKL150. It was also clearly noted in Debian's intel-microcode changelog on May 15: http://metadata.ftp-master.debian.org/changelogs/non-free/i/intel-microcode/intel-microcode_3.20170511.1_changelog http://metadata.ftp-master.debian.org/changelogs/non-free/i/... They absolutely should have followed up to the OCaml people's support ticket. But sloppy followup is an issue that every large project encounters.

fotcorn 9y ago

The latest intel-microcode package from Ubuntu 16.04 does not fix the problem. I installed the same package from Ubuntu 17.10 [0] which fixes the problem. You can check your system with the script linked in the mailing list thread [1]. [0] https://packages.ubuntu.com/en/artful/amd64/intel-microcode/download https://packages.ubuntu.com/en/artful/amd64/intel-microcode/... [1] https://lists.debian.org/debian-devel/2017/06/msg00309.html https://lists.debian.org/debian-devel/2017/06/msg00309.html

edwinksl 9y ago
Launchpad bug report for this: https://bugs.launchpad.net/ubuntu/+source/intel-microcode/+bug/1700373 https://bugs.launchpad.net/ubuntu/+source/intel-microcode/+b...
arde 9y ago
Indeed, the latest Intel microcode published for Ubuntu 16.04 is the ancient 20151106 [1]. Later Ubuntu releases do have more recent microcode packages [2]. I cannot understand why they left out 16.04 there. So much for LTS, it seems. This recently came to my attention while debugging some increasingly frequent lockups, which took me a solid week of eliminating all seemingly more likely causes (VirtualBox, nVidia driver, faulty RAM, etc). In the end I found the culprit while digging into the Intel Specification updates: my Core i7-5820K (and most other Haswell-E and Broadwell processors) has a bug when leaving package C-states, and the only workaround is to disable C-states above level 1. Timely updated microcode, which applies this workaround, would have saved me my week. [1] https://launchpad.net/ubuntu/xenial/+source/intel-microcode https://launchpad.net/ubuntu/xenial/+source/intel-microcode [2] https://launchpad.net/ubuntu/+source/intel-microcode/+changelog https://launchpad.net/ubuntu/+source/intel-microcode/+change...
- rlpb 9y ago
  > Indeed, the latest Intel microcode published for Ubuntu 16.04 is the ancient 20151106. By ancient, perhaps you mean the version that was current at the time 16.04 shipped? > I cannot understand why they left out 16.04 there. So much for LTS, it seems. See https://wiki.ubuntu.com/StableReleaseUpdates https://wiki.ubuntu.com/StableReleaseUpdates. The point of an LTS (or any stable release, for that matter) is that it doesn't change by default. For those who want to keep everything up-to-date, Ubuntu ships a new release on a six month cadence. If you choose not to use that, then you shouldn't be surprised when things aren't updated, since that's exactly what you opted in to. The microcode package may warrant an exception, however, and we have a bug to track that. It's tricky because without the source we cannot pick apart what changed, or determine whether any changes meet our update policy. We have to be careful. Sooner or later some user will inevitably come along to tell us that a microcode update broke things, and ask why we didn't fulfill our LTS promise by not changing it.

tyingq 9y ago

There's a perl script on the debian mailing list that digs a bit deeper and tells you if you're affected in the first place, if you're affected but patched already, affected but have HT disabled, etc. https://lists.debian.org/debian-user/2017/06/msg01011.html https://lists.debian.org/debian-user/2017/06/msg01011.html

fattire 9y ago
I ported this to bash since I have a chromebook w/o perl and (as for right now) the fs is read-only, so I just piped the script to it and sure enough my brand-new Samsung Chromebook Pro appears to be vulnerable, though apparently patchable. Details and if you want the script I link to it from here: https://forum.xda-developers.com/hardware-hacking/chromebooks/samsung-chromebook-pro-asus-c302ca-t3627253 https://forum.xda-developers.com/hardware-hacking/chromebook... - don't judge my shitty bash skills. ft

Syzygies 9y ago

In my experience with parallel code written in Haskell, hyper-threading offers only a very mild speedup, perhaps 10%. It is essentially an illusion, a logical convenience. (How long does it take to complete a parallel task on a dedicated machine? Four cores with hyper-threading off has nearly the performance of eight virtual cores with hyper-threading on.) Many people have neither the interest nor the hardware access to overclock, and these processors have less overclocking headroom than earlier designs. Nevertheless, the hyper-threading hardware itself generates heat, restricting the overclocking range for given cpu cooling hardware. In this case, turning off hyper-threading pays for itself, because one can then overclock further, overtaking any advantage to hyper-threading.

barrkel 9y ago
It depends on what resources your code uses on-chip. If all threads are contending on the same resources, then you won't see a speedup; if they're using different resources, hyperthreading can increase throughput significantly. I've seen hyperthreading give me the equivalent of 50% of another CPU, particularly when I'm running multiple CPU-bound processes concurrently (so they're not executing the same code at the same time in some kind of parallel operation, and certainly aren't bound on synchronization primitive overheads).
- Syzygies 9y ago
  That makes sense. I'm a mathematician, and my experience is with pure computations, homogeneous across each (virtual) core.
esaym 9y ago
It normally helps "some", and rarely hurts performance. So you might as well enable it.
- ploxiln 9y ago
  I worry about hyperthreading hurting worst-case latency (since a thread might be assigned to run on a virtual core which does not work as fast as expected).

ncrmro 9y ago

Just got the 2017 no touchbar 13 macbook pro with the kaby lake i7. Should I be worried, can I even disable HT with mac. And presumably the update will be provided so the whole laptop is still ok? I've been using the thunderbolt 3 dock with two external monitors and occasionally get a little glitch prolly loose cable I think. I've downloaded the bitcoin blockchain, done quite a bit of work in pycharm + chrome, multiple projects, flow and webpack in the background and haven't had any sort of crashes tho.

pedrocr 9y ago

Here's how to fix it on a Thinkpad on Linux. I've got a T460s and checked with the script[1] that it was indeed affected. The Debian instructions said to update your BIOS before updating the microcode package so I went to the model support page[2] to the BIOS/UEFI section and downloaded the "BIOS Update (Bootable CD)" one. The changelog included microcode updates so it looked promising[3]. To get the ISO onto a usb drive I did the following: $ geteltorito n1cur14w.iso > eltorito-bios.iso # provided by the genisoimage package on Ubuntu $ sudo dd if=eltorito-bios.iso of=/dev/sdXXX # replace with your usb drive with care to not write over your disk I then had a bootable USB drive that I ran by rebooting the computer, pressing Enter and then F12 to get to the boot drive selection and selecting the USB. From then it's just following the options it gives you. It's basically pressing 2 to go into the update and then pressing Y and Enter a few times to tell it you really want to do it. After that just let it reboot a few times and the update is done. After booting again the same test script[1] now said I had an affected CPU but new enough microcode. [1] https://lists.debian.org/debian-user/2017/06/msg01011.html https://lists.debian.org/debian-user/2017/06/msg01011.html [2] http://pcsupport.lenovo.com/pt/en/products/laptops-and-netbooks/thinkpad-t-series-laptops/thinkpad-t460s/downloads http://pcsupport.lenovo.com/pt/en/products/laptops-and-netbo... [3] https://download.lenovo.com/pccbbs/mobiles/n1cur14w.txt https://download.lenovo.com/pccbbs/mobiles/n1cur14w.txt

geogriffin 9y ago

has anyone affected by this bug tried using a kernel configured with hyperthreading support disabled? would that work?

mjw1007 9y ago

It's painful to have to read text like « select Intel Pentium processor models ». If Intel used marketing names that were more closely related to technical reality, then when something like this happens they wouldn't have so many customers finding themselves in the "maybe I'm affected by this horrid bug" box.

asow92 9y ago

So what does this mean for the thousands of new MacBook Pro 2016/2017 owners out there?

coldtea 9y ago
Nothing, since they've been running their laptops without issues (it would have been all over the news if it was some widespread issue) for 2+ years. At some point in the near future Apple will package the microcode fix in an update, and that will be it.
- thoughtsimple 9y ago
  Probably already done since this was fixed in April/May by Intel. There has been at least one update since then.
- donkeyd 9y ago
  I've run into an issue where switching users causes a crash on my 2016 MBP. Many more people are having this issue, according to the Google. Also, up until the end of 2016, Apple didn't use Skylake processors, so it wouldn't be 2+ years.

zzalpha 9y ago

Charming. I picked up a 5th Gen X1 Carbon configured with a Kaby Lake processor, and apparently there's no way to disable hyperthreading in the BIOS, and according to Intel's errata, no fix available yet. Oh well... so far the machine (running Windows 10) has been stable minus one or two random lockups in 2 months of heavy usage which could be attributed to this. Guess I wait...

elnik 9y ago

My CPU (6th generation i5) died last week. RIP. I installed debian 9, installed virtualbox, vagrant, setup a clean development machine for myself, everything took 4 hours to finish. I reboot the virtual machine, and boom, there was a kernel panic which I sadly don't remember exactly / didn't take a picture of. After I rebooted the machine, and opened terminal, the system froze. The cursor wouldn't move. Reboot again, motherboard has a CPU fail/undetected light on. Couldn't get it to boot after that. I am both sad and relieved that bad stuff exists, but it's being patched to prevent proliferating. I sincerely hope I'll get a replacement from Intel.

broknbottle 9y ago
CPUs rarely die, unless you're OCing or PSU went bad and took things out, I am willing to bet your MoBo is the part that is bad.

luckydude 9y ago

That's a really nicely done announcement. Simple, to the point, no drama, all the info you could want, scripts to figure out your processor, etc. Well done Debian folks!

ComputerGuru 9y ago

So, serious question: If the microcode "fix" for this ends up disabling HT, how does one get a refund not just for the CPU but for the $3k laptop I spec'd around it? Without needing to sue? This isn't a hypothetical; what did Intel do when the only fix for broken functionality was to disable TSX entirely?

mixmastamyk 9y ago
I remember the pentium bug in the mid 90s, they actually shipped out replacement processors. Doubt that could be pulled off on laptops. Perhaps a microcode update can work around it.
- senectus1 9y ago
  Given my Surface Book got a 1/10 repair-ability score on ifixit... I dont think they'll just replace the chip :-P
paulddraper 9y ago
The same for any part of your computer, I imagine. What happens when your laptop's display has frequently broken pixels?
- sqldba 9y ago
  > What happens when your laptop's display has frequently broken pixels? Well they wait for 6 or more before they say it's out of spec

bleair 9y ago

When intel had the floating point division hardware bug they recalled chips. https://en.wikipedia.org/wiki/Pentium_FDIV_bug https://en.wikipedia.org/wiki/Pentium_FDIV_bug I wonder if intel will do something like that again or if the industry as a whole is more tolerant of unreliable / buggy behavior and will just live with it. Examples of Apple just telling people that the poor reception strength was their own fault / changing software to hide problems / etc.

aarongolliver 9y ago
If you read the link you'll see they've fixed it with a microcode update.

itsoggy 9y ago

When HT first started appearing on P4 chips I was looking after NetWare, 2K and XP boxes, they would freak out with HT enabled all kinds of oddities, I suspect most because of the OS's not fully supporting it. To this day I disable it by reflex on everything!

cJ0th 9y ago

This is just great! Just yesterday I got a new laptop with a skylake processor. Now I wonder whether I experienced that bug today as I got kicked out of dosbox (on debian) for no apparent reason. In the config file I had to change the value of 'core' in the cpu-section from 'automatic' to 'normal'. Could be something entirely different but it is a funny timing.

whitebowen 9y ago

Do you need a professional hacker, contact cipherhacker1@GMAIL.COM Our service includes Change University grades Facebook, twitter, IG hack Email hack icloud and phone hack * grades change Wipe criminal records Wipe credit card debt MasterCard's/visa cards Bank account *Data base hack and lot more hacking services in general Among other customized services...all this are at all great rate. Results guaranteed. Contact us at cipherhacker1@GMAIL.COM

whitebowen 9y ago

joshschreuder 9y ago

How does one get new microcode on Windows? Is that what the Intel Chipset drivers are? And is the microcode fix available for non-Linux systems yet?

aarongolliver 9y ago
Windows update
- tempestn 9y ago
  Has this already been patched via Windows Update? I didn't even know WU could do CPU microcode patches.

whitebowen 9y ago

age_bronze 9y ago

I would've expected at least an example assembly code reproducing the bug? How was it not discovered before, but only with the OCaml compiler? They say "unexpected behavior", does this mean that code compiled with this can give incorrect results? Can this have any security implication? How much code was compiled with similar patterns? Can the problem reproduced with any JIT compiler? We need to know what can cause this, maybe compiled and working code already contains such patterns waiting to be abused...

wfunction 9y ago

A little off-topic, but does anybody know of any hacky ways to disable hyper-threading (on Haswell if it matters) if the firmware doesn't provide the option?

spektom 9y ago

I wonder how this issue affects cloud providers?

Traubenfuchs 9y ago

Are those microcode updates deployed via windows (10) update?

paines 9y ago

I have a Skylake mobile CPU (i7-6700hq) and it pretty much rocks with Ubuntu 17. Also the system is stable and fast. Under heavy load, e.g. games the system is stable. Compiling a big(>10000 modules) C++ project via ninja/cmake under Qt Creator hangs the system resporducibly after ~15 minutes. I wonder now if this broken hyperthreading could be such sideffect.

herpderperator 9y ago

If anyone on Windows wishes to update their CPU microcode without waiting for Microsoft to push it out via Windows Update, you can use this tool from VMware https://labs.vmware.com/flings/vmware-cpu-microcode-update-driver https://labs.vmware.com/flings/vmware-cpu-microcode-update-d... which can update microcode as well. Windows stores its microcode in C:\Windows\System32\mcupdate_GenuineIntel.dll which is a proprietary binary file and you can't simply replace it with Intel's microcode.dat file (which is ASCII text), so you have to use a third-party tool such as VMware's one. Simply: 1. Download and extract the zip file in the first paragraph 2. Modify the install.bat file so that the line which reads `for %%i IN (microcode.dat microcode_amd.bin microcode_amd_fam15h.bin) DO (` only contains the microcode.dat parameter (since you obviously don't have an AMD CPU, and the tool is made for both) 3. Download and extract microcode.dat from Intel's website (https://downloadcenter.intel.com/download/26798/Linux-Processor-Microcode-Data-File https://downloadcenter.intel.com/download/26798/Linux-Proces...) and place it into the same directory as the VMware tool 4. Run install.bat with admin privileges 5. Hit cancel when it tells you that the AMD microcode files are missing, and you're done The CPU microcode will be updated immediately (yes, while Windows is running.) The service will also run on each boot and update your CPU microcode, since microcode updates are only temporary and are lost each time you restart. You can check Event Viewer for entries from `cpumcupdate` to see what it has done. It's advised to run a tool to view the microcode version before installing (such as HWiINFO64) so you can re-run the tool after installing and confirming that the version has changed. I have done this and it works as described. I went from 0x74 to 0xba as shown by the μCU field in HWiNFO64, and I have an i7-6700k.

wscott 9y ago

Has anyone benchmarked one of these machines before and after applying this microcode update? The options in microcode are rather limited and all are likely to have performance impacts. This is likely disabling functionally to avoid this case. I would hope the patch is smart enough to not apply if threading is not enabled, but who knows.

jakeogh 9y ago
Would a performance hit go unnoticed? Usually (always?) it's not a ROM update, the encrypted microcode blob is loaded into the CPU by the OS on every boot via CONFIG_MICROCODE. some linkrot: http://imgur.com/a/z1uLv http://imgur.com/a/z1uLv

jwildeboer 9y ago

I do wonder though, why didn't Debian maintainers pick up the microcode updates when they were made available by Intel? Why did it need a wink from the Ocaml people for them to note? Or am I missing something?

msimpson 9y ago

The late 2016 Razer Blade uses the i7-6700HQ which is specifically a Family 6, Model 94, Stepping 3 processor. I wonder if a microcode update would solve some of the various issues I have in Windows.

karussell 9y ago

What is the probability for this to happen? Or how could I estimate the time it takes for random code to hit this bug at least once with a probability of over 90%?

ManyEthers 9y ago

Intel's communication is incredibly poor. Errata exist for all CPUs but this one is quite important and resulted in no proper public communication it seems.

Magnificents 9y ago

The poor guys from OCaml who found the bug. Imagine how much debugging it takes to find such an issue and narrow it down to the precise register sequence. I guess since it’s a hyper threading bug it even depends on multiple threads doing certain things at the same time. Usually you trust your CPU to execute code properly.

convefefe 9y ago

One of many typical erratum... nothing to see here, been patched months ago. Most people are unlikely to ever encounter it even if unpatched.

peter_retief 9y ago

If I run $grep name /proc/cpuinfo | sort -u model name : Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz then $cat /proc/cpuinfo | grep ht Definitely there under flags Should I be concerned?

itchyouch 9y ago
Your processor isn't affected. It would be i[357]-[67]xxx processors only. But a new computer may serve you well...

ericfrederich 9y ago

Is it fixed or not? Beginning of post says to disable hyperthreading but then goes on to say Intel fixed this with a microcode update.