15 ms·

How I found a bug in Intel Skylake processors (2017)

295 points by vinnyglennon 5y ago

facorreia 5y ago
2017.
deleted 5y ago
[deleted]
wging 5y ago
Previous submission: https://news.ycombinator.com/item?id=14686277 https://news.ycombinator.com/item?id=14686277 (This is not a complaint; I found the post interesting.)
- dang 5y ago
  Thanks! Macroexpanded: I found a bug in Intel Skylake processors - https://news.ycombinator.com/item?id=14686277 https://news.ycombinator.com/item?id=14686277 - July 2017 (99 comments)
deleted 5y ago
[deleted]
SavantIdiot 5y ago
This is a scary place to be: the top-level debug resource for a major project. It took almost two years to resolve, but was already known as SKL150. Looking at the clang vs. gcc assembly without knowledge of SKL150 would be literally impossible to debug. GCC -O1 vs -O2 is a clue, but even with the asm diffs, wth? Again, scary.
- tinus_hn 5y ago
  The world is a scary place; this is basically the same as rowhammer which is an issue in computers shipped today.
  - woodruffw 5y ago
    Unless I'm misunderstanding what you mean, this isn't really like rowhammer at all -- it's a uarch/ucode bug, which is effectively a programming error within the CPU. Rowhammer is a physical flaw in how memory cells in DRAM are laid out, one that can be triggered by memory access patterns independent of CPU architecture and microarchitecture. (There are also hundreds of errata like this one in every CPU generation. They're usually not easy to exploit, since they cause system instability rather than disclosing secret material or allowing unintended code execution.)
    - zsmi 5y ago
      > Rowhammer is a physical flaw in how memory cells in DRAM are laid out It's not really a flaw, more like a consequence of how memory cells are laid out. I mean most people want lots of bits in their DRAM. Maximizing this parameter necessitates that some will be in close proximity.
      - woodruffw 5y ago
        To my (non-EE) mind, the flaw is the electrical leakage between the cells. Tight packing is a consequence of economic forces, but I assume there are also technical solutions that allow for tight packing (but either offset the performance or cost gains). Is that assumption wrong? (Genuinely asking!)
        zsmi 5y ago
        There was a good paper on it in 2014. [1] They describe the RowHammer attack as: opening and closing (activation and precharge) a DRAM row (aggressor row) at a high enough rate (hammering) such that it can cause bit-flips in physically nearby rows (victim row). Colloquially, it's basically a change in voltage in one place can indirectly cause a change in voltage in another place via capacitive coupling. Capacitance increases proportional to the inverse of the separating distance so only in recent years have things shrunk to the size that makes it an issue. Since having less bits in DRAM is basically not an option most mitigation techniques that I know of remove the possibility of hammering: possibilities include the OS, memory system controller, or DRAM controller changes. [1] https://users.ece.cmu.edu/~yoonguk/papers/kim-isca14.pdf https://users.ece.cmu.edu/~yoonguk/papers/kim-isca14.pdf
CalChris 5y ago
Debian announcement https://lists.debian.org/debian-devel/2017/06/msg00308.html https://lists.debian.org/debian-devel/2017/06/msg00308.html Ahrefs writeup https://tech.ahrefs.com/skylake-bug-a-detective-story-ab1ad2beddcd https://tech.ahrefs.com/skylake-bug-a-detective-story-ab1ad2... The Intel spec update still labels SKL150 as No Fix but there is a microcode update available. Dunno exactly what to make of that distinction. https://www.intel.com/content/www/us/en/processors/core/desktop-6th-gen-core-family-spec-update.html https://www.intel.com/content/www/us/en/processors/core/desk... Can an x86 program detect whether this update has been applied? Can a Linux process set a DONT_HYPERTHREAD_ME_BRO bit?
- BeeOnRope 5y ago
  It was "fixed" in a microcode update by disabling the loop stream buffer (LSD) which is a special mode of operation for very small loops where the instruction decoders and uop cache in the CPU are shut down and the loop runs directly out of a small cache*. Since the problem arose only when the LSD was being used, in combination with hyperthreading and high byte register use, this effectively avoids the problem. Of course, disabling the LSD has some costs: CPUs use more power and some loops are slower (though some are faster). These updates are usually applied silently without user consent, so you might quite surprised to find out that after a reboot your computation kernel suddenly draws more power or has slowed down or sped up. > Can an x86 program detect whether this update has been applied? Can a Linux process set a DONT_HYPERTHREAD_ME_BRO bit? Yes. One way would be to check the microcode version (available in /proc/cpuinfo on Linux, among other places), since the version that introduced this fix is known. Another way would be to run a small loop known to fit in the LSD and then check a performance counter event which counts uops delivered from the LSD, like lsd.uops. This counter is always zero when the LSD is disabled (or realistically you could just run any substantial code and check the counter since you always have some non-neglible portion of the uops coming from the LSD). This is how I check it from the command line in practice. Finally, if you don't have easy access to the counters, you could create a loop that has a significant performance difference depending on whether it is coming from the LSD or not. For example, a loop that crosses a 32-byte boundary will run 2 or more cycles when using the decoder or uop cache, but could run in 1 cycle in the LSD. Timing such a loop would give you a strong indication about whether the LSD is enabled. --- * Specifically, the cache used is not a dedicated one, but rather the IDQ (decoded instruction queue) is reused. This queue holds uops and is normally fed by the decoders or the uop cache on one end, and which feeds the allocation/rename engine on the other. In LSD mode, this queue stops being a queue and is instead used as a kind of cache with the loop operations "locked down" in the queue and just repeatedly replayed.
lordnacho 5y ago
The problem with bugs deep in the stack is that it is really time consuming to establish that they are in fact as deep as they are. I wrote a Swift iOS app once, and came across an issue with one of the collection classes. Of course, nobody thinks that the Swift libs will be wrong as a first guess. So I worked through a number of hypotheses about my own code, slowly stripping out pieces that I thought might contain an error. And then combinations. I also tried reducing the number of entries just to simplify the logs. This worked, but of course you are not going to think that there's a library bug affecting collections with size > 16, and it wasn't actually a theory until I randomly decided to reduce the n. I also discovered that it worked just fine on release but not debug, so I thought maybe I have some race condition. More and more stripping down occurred, until I eventually gave up using my own project and just started a new one just to see about the collection class. I did it for the sake of being thorough, rather than actually thinking the lib had a bug in its debug implementation. But lo and behold, when I managed to make it reproducible and put it on SO, someone from Apple acknowledged that they could also see it, and they fixed it. Naturally if I'd gone direct to testing the lib I'd have saved a huge amount of time, but I guess that's the tradeoff from the most sensible heuristic: test your own code first, the bug is there.
- yjftsjthsd-h 5y ago
  Reminds me of: "It Is Never a Compiler Bug Until It Is" (https://r6.ca/blog/20200929T023701Z.html https://r6.ca/blog/20200929T023701Z.html , https://news.ycombinator.com/item?id=24636326 https://news.ycombinator.com/item?id=24636326). The bottom of the modern stack is really reliable, until it isn't;)
  - Smoosh 5y ago
    Not just "the modern stack". I work mainframes and always felt the IBM-supplied environment (compilers, transaction processing systems, databases) was rock solid. Then one day I discovered APARs were a thing. https://www.ibm.com/support/pages/open-apars-ibm-products-available-web https://www.ibm.com/support/pages/open-apars-ibm-products-av...
- cesarb 5y ago
1432132143 5y ago
really guys GFY you know what OEMs do, they disable many features every time got some new bug. i.e undervolting now my thinkbook fan is always on on my laptop 30* fan is on 29* fan is on can't even undervold my cpu now. Realy thx
dimitrios1 5y ago
Apologies if this is off topic -- but I am constantly impressed at some of the things I find that come from inria.fr. I first came across them when learning OCaml. Seems to be a top notch university.
- woodruffw 5y ago
  Inria is a research institute, not a university. But they do indeed do excellent work!
bruce343434 5y ago
The link called "6th Generation Intel® Processor Family - Specification Update." 404's
Decabytes 5y ago
I’m glad I’m just a pleb programmer, who never has done anything so complicated that it would expose processor errata. And even if I did, I wouldn’t have the expertise to even figure it out.
- dfox 5y ago
  The issue there is that the hardware is full of totally absurd bugs. If you target PC-like userspace or one of the two major mobile platforms it is somebody else's job to shield you from that. In general CPU level bugs are somewhat rare, but every single platform vendor had shipped some kind of silicon that contains peripherals that do not work as documented and only by chance work with the reference driver implementation.
- brokenmachine 5y ago
  Welcome to the 99.999999999%.
bjarneh 5y ago
> Binary search always fails? “The Java compiler is acting funny today!” :-)
userbinator 5y ago
"gcc/clang/icc/msvc won't usually issue the affected opcode pattern and it ends up being rare. SKL150 - Short loops using both the AH/BH/CH/DH registers and the corresponding wide register may result in unpredictable system behavior." I think Intel should regression-test its CPUs using the decades of demoscene productions out there, especially those in the extreme-size-optimisation categories; testing with almost exclusively "mainstream" compiler output is IMHO a bad idea and a step down the path to "warranty void if VLC is used" (https://news.ycombinator.com/item?id=7205759 https://news.ycombinator.com/item?id=7205759 )
13of40 5y ago
> More experienced programmers know very well that the bug is generally in their code: occasionally in third-party libraries; very rarely in system libraries This was the bane of my existence when I worked on testing Windows years ago. New SDETs almost invariably fell into the trap of assuming any automation error was a "test bug" instead of a bug in OS code, even if the OS code in question was written last week.
pabs3 5y ago
This reminds me of sandsifter and Google's Silifuzz: https://github.com/xoreaxeaxeax/sandsifter https://github.com/xoreaxeaxeax/sandsifter https://github.com/google/fuzzing/blob/master/docs/silifuzz.pdf https://github.com/google/fuzzing/blob/master/docs/silifuzz.... https://www.phoronix.com/scan.php?page=news_item&px=Google-SiliFuzz-CPU-Defects https://www.phoronix.com/scan.php?page=news_item&px=Google-S...

295 points by vinnyglennon 5y ago

facorreia 5y ago

2017.

deleted 5y ago

[deleted]

wging 5y ago

Previous submission: https://news.ycombinator.com/item?id=14686277 https://news.ycombinator.com/item?id=14686277 (This is not a complaint; I found the post interesting.)

dang 5y ago
Thanks! Macroexpanded: I found a bug in Intel Skylake processors - https://news.ycombinator.com/item?id=14686277 https://news.ycombinator.com/item?id=14686277 - July 2017 (99 comments)

deleted 5y ago

[deleted]

SavantIdiot 5y ago

This is a scary place to be: the top-level debug resource for a major project. It took almost two years to resolve, but was already known as SKL150. Looking at the clang vs. gcc assembly without knowledge of SKL150 would be literally impossible to debug. GCC -O1 vs -O2 is a clue, but even with the asm diffs, wth? Again, scary.

tinus_hn 5y ago
The world is a scary place; this is basically the same as rowhammer which is an issue in computers shipped today.
- woodruffw 5y ago
  Unless I'm misunderstanding what you mean, this isn't really like rowhammer at all -- it's a uarch/ucode bug, which is effectively a programming error within the CPU. Rowhammer is a physical flaw in how memory cells in DRAM are laid out, one that can be triggered by memory access patterns independent of CPU architecture and microarchitecture. (There are also hundreds of errata like this one in every CPU generation. They're usually not easy to exploit, since they cause system instability rather than disclosing secret material or allowing unintended code execution.)
  - zsmi 5y ago
    > Rowhammer is a physical flaw in how memory cells in DRAM are laid out It's not really a flaw, more like a consequence of how memory cells are laid out. I mean most people want lots of bits in their DRAM. Maximizing this parameter necessitates that some will be in close proximity.
    - woodruffw 5y ago
      To my (non-EE) mind, the flaw is the electrical leakage between the cells. Tight packing is a consequence of economic forces, but I assume there are also technical solutions that allow for tight packing (but either offset the performance or cost gains). Is that assumption wrong? (Genuinely asking!)
      - zsmi 5y ago
        There was a good paper on it in 2014. [1] They describe the RowHammer attack as: opening and closing (activation and precharge) a DRAM row (aggressor row) at a high enough rate (hammering) such that it can cause bit-flips in physically nearby rows (victim row). Colloquially, it's basically a change in voltage in one place can indirectly cause a change in voltage in another place via capacitive coupling. Capacitance increases proportional to the inverse of the separating distance so only in recent years have things shrunk to the size that makes it an issue. Since having less bits in DRAM is basically not an option most mitigation techniques that I know of remove the possibility of hammering: possibilities include the OS, memory system controller, or DRAM controller changes. [1] https://users.ece.cmu.edu/~yoonguk/papers/kim-isca14.pdf https://users.ece.cmu.edu/~yoonguk/papers/kim-isca14.pdf

CalChris 5y ago

Debian announcement https://lists.debian.org/debian-devel/2017/06/msg00308.html https://lists.debian.org/debian-devel/2017/06/msg00308.html Ahrefs writeup https://tech.ahrefs.com/skylake-bug-a-detective-story-ab1ad2beddcd https://tech.ahrefs.com/skylake-bug-a-detective-story-ab1ad2... The Intel spec update still labels SKL150 as No Fix but there is a microcode update available. Dunno exactly what to make of that distinction. https://www.intel.com/content/www/us/en/processors/core/desktop-6th-gen-core-family-spec-update.html https://www.intel.com/content/www/us/en/processors/core/desk... Can an x86 program detect whether this update has been applied? Can a Linux process set a DONT_HYPERTHREAD_ME_BRO bit?

BeeOnRope 5y ago
It was "fixed" in a microcode update by disabling the loop stream buffer (LSD) which is a special mode of operation for very small loops where the instruction decoders and uop cache in the CPU are shut down and the loop runs directly out of a small cache*. Since the problem arose only when the LSD was being used, in combination with hyperthreading and high byte register use, this effectively avoids the problem. Of course, disabling the LSD has some costs: CPUs use more power and some loops are slower (though some are faster). These updates are usually applied silently without user consent, so you might quite surprised to find out that after a reboot your computation kernel suddenly draws more power or has slowed down or sped up. > Can an x86 program detect whether this update has been applied? Can a Linux process set a DONT_HYPERTHREAD_ME_BRO bit? Yes. One way would be to check the microcode version (available in /proc/cpuinfo on Linux, among other places), since the version that introduced this fix is known. Another way would be to run a small loop known to fit in the LSD and then check a performance counter event which counts uops delivered from the LSD, like lsd.uops. This counter is always zero when the LSD is disabled (or realistically you could just run any substantial code and check the counter since you always have some non-neglible portion of the uops coming from the LSD). This is how I check it from the command line in practice. Finally, if you don't have easy access to the counters, you could create a loop that has a significant performance difference depending on whether it is coming from the LSD or not. For example, a loop that crosses a 32-byte boundary will run 2 or more cycles when using the decoder or uop cache, but could run in 1 cycle in the LSD. Timing such a loop would give you a strong indication about whether the LSD is enabled. --- * Specifically, the cache used is not a dedicated one, but rather the IDQ (decoded instruction queue) is reused. This queue holds uops and is normally fed by the decoders or the uop cache on one end, and which feeds the allocation/rename engine on the other. In LSD mode, this queue stops being a queue and is instead used as a kind of cache with the loop operations "locked down" in the queue and just repeatedly replayed.

lordnacho 5y ago

The problem with bugs deep in the stack is that it is really time consuming to establish that they are in fact as deep as they are. I wrote a Swift iOS app once, and came across an issue with one of the collection classes. Of course, nobody thinks that the Swift libs will be wrong as a first guess. So I worked through a number of hypotheses about my own code, slowly stripping out pieces that I thought might contain an error. And then combinations. I also tried reducing the number of entries just to simplify the logs. This worked, but of course you are not going to think that there's a library bug affecting collections with size > 16, and it wasn't actually a theory until I randomly decided to reduce the n. I also discovered that it worked just fine on release but not debug, so I thought maybe I have some race condition. More and more stripping down occurred, until I eventually gave up using my own project and just started a new one just to see about the collection class. I did it for the sake of being thorough, rather than actually thinking the lib had a bug in its debug implementation. But lo and behold, when I managed to make it reproducible and put it on SO, someone from Apple acknowledged that they could also see it, and they fixed it. Naturally if I'd gone direct to testing the lib I'd have saved a huge amount of time, but I guess that's the tradeoff from the most sensible heuristic: test your own code first, the bug is there.

yjftsjthsd-h 5y ago
Reminds me of: "It Is Never a Compiler Bug Until It Is" (https://r6.ca/blog/20200929T023701Z.html https://r6.ca/blog/20200929T023701Z.html , https://news.ycombinator.com/item?id=24636326 https://news.ycombinator.com/item?id=24636326). The bottom of the modern stack is really reliable, until it isn't;)
- Smoosh 5y ago
  Not just "the modern stack". I work mainframes and always felt the IBM-supplied environment (compilers, transaction processing systems, databases) was rock solid. Then one day I discovered APARs were a thing. https://www.ibm.com/support/pages/open-apars-ibm-products-available-web https://www.ibm.com/support/pages/open-apars-ibm-products-av...
cesarb 5y ago

1432132143 5y ago

really guys GFY you know what OEMs do, they disable many features every time got some new bug. i.e undervolting now my thinkbook fan is always on on my laptop 30* fan is on 29* fan is on can't even undervold my cpu now. Realy thx

dimitrios1 5y ago

Apologies if this is off topic -- but I am constantly impressed at some of the things I find that come from inria.fr. I first came across them when learning OCaml. Seems to be a top notch university.

woodruffw 5y ago
Inria is a research institute, not a university. But they do indeed do excellent work!

bruce343434 5y ago

The link called "6th Generation Intel® Processor Family - Specification Update." 404's

Decabytes 5y ago

I’m glad I’m just a pleb programmer, who never has done anything so complicated that it would expose processor errata. And even if I did, I wouldn’t have the expertise to even figure it out.

dfox 5y ago
The issue there is that the hardware is full of totally absurd bugs. If you target PC-like userspace or one of the two major mobile platforms it is somebody else's job to shield you from that. In general CPU level bugs are somewhat rare, but every single platform vendor had shipped some kind of silicon that contains peripherals that do not work as documented and only by chance work with the reference driver implementation.
brokenmachine 5y ago
Welcome to the 99.999999999%.

bjarneh 5y ago

> Binary search always fails? “The Java compiler is acting funny today!” :-)

userbinator 5y ago

"gcc/clang/icc/msvc won't usually issue the affected opcode pattern and it ends up being rare. SKL150 - Short loops using both the AH/BH/CH/DH registers and the corresponding wide register may result in unpredictable system behavior." I think Intel should regression-test its CPUs using the decades of demoscene productions out there, especially those in the extreme-size-optimisation categories; testing with almost exclusively "mainstream" compiler output is IMHO a bad idea and a step down the path to "warranty void if VLC is used" (https://news.ycombinator.com/item?id=7205759 https://news.ycombinator.com/item?id=7205759 )

13of40 5y ago

> More experienced programmers know very well that the bug is generally in their code: occasionally in third-party libraries; very rarely in system libraries This was the bane of my existence when I worked on testing Windows years ago. New SDETs almost invariably fell into the trap of assuming any automation error was a "test bug" instead of a bug in OS code, even if the OS code in question was written last week.

pabs3 5y ago

This reminds me of sandsifter and Google's Silifuzz: https://github.com/xoreaxeaxeax/sandsifter https://github.com/xoreaxeaxeax/sandsifter https://github.com/google/fuzzing/blob/master/docs/silifuzz.pdf https://github.com/google/fuzzing/blob/master/docs/silifuzz.... https://www.phoronix.com/scan.php?page=news_item&px=Google-SiliFuzz-CPU-Defects https://www.phoronix.com/scan.php?page=news_item&px=Google-S...