8 ms·

Mozilla finds CPU bug (bad store forwarding) in Samsung Galaxy S20

246 points by kens 3y ago

LispSporks22 3y ago
The only thing better than being able to blame the compiler for something, is being able to blame the CPU
- print_goto_ten 3y ago
  Great tshirt
- peheje 3y ago
  Saving that quote.
- amelius 3y ago
  But how are you going to fix it then?
  - wongarsu 3y ago
    Change your code to not trigger the bug, and leave that workaround there for the next decade or so until the buggy CPU generation isn't relevant anymore. Or hope for microcode updates to fix the bug at the CPU level.
    - amelius 3y ago
      That's my point ... it's a lot harder to do this than to just fix it in your own code.
  - JonChesterfield 3y ago
    You change the compiler to carefully step around the stuff that doesn't work in the hardware.
  - pajko 3y ago
    Compilers can be patched to automatically do some masking in the code generation step after every LDRB instruction in this case, when compiling for that specific CPU. For example, the Blackfin DSPs have many similar bugs which are fixed this way. Some of which the fixes done automatically, while others could be disabled for performance critical code which has been implemented while keeping the various issues in mind, like -mcsync-anomaly and -mspecld-anomaly. Certain MIPS CPUs also have incorrectly implemented instructions and the -mfix-* compiler parameters could be used to enable workarounds. JIT compilers can do the same.
jeffbee 3y ago
This is where the vocal minority of HN posters who are in a snit about telemetry are wrong. This is the kind of issue you can discover at global scale, but only if you collect the data.
- n4r9 3y ago
  It looks like this could have been discovered by simply having (anonymised) crash report telemetry? I don't know about others, but that sort of telemetry doesn't sound so bad.
  - Nextgrid 3y ago
    It's very hard to have useful anonymized telemetry. In this case, you actually need at least a partial memory or register dump to understand what's going on, but those can contain sensitive data that the failing code was processing. Telemetry doesn't have to be 100% anonymized - it can (and in some cases needs) to contain PII, just needs to be collected respectfully and transparently. Ask the user before sending something, and let the user review it. The user can then make their own decision.
    - bpye 3y ago
      > Ask the user before sending something, and let the user review it. The user can then make their own decision. I under The sentiment, but say you were sending a memory dump as part of your crash reporting. How is a user going to review that?
- mixedbit 3y ago
  Such crash data can be collected with an old fashion 'Firefox has crashed, send a problem report to Mozilla [OK] [Cancel]' dialog, which is much less privacy-intrusive than telemetry.
  - ksenzee 3y ago
    Right, and Firefox for Android does in fact collect its crash data like this.
garganzol 3y ago
When I was a kid, I always suspected a compiler or CPU bug when my program did not behave as expected. I heard similar stories on more than several occasions from other people, so being naïve must be a common thing among us software engineers when we make our first steps. Kudos to Mozilla, their finding is a real CPU bug.
- pengaru 3y ago
  For ages in *nixland the phrase "select is not broken" has been an aphorism for such situations. I'm not sure of its origin, but it's referenced in The Pragmatic Programmer: > Tip #33, pg. 95: > > “select” Isn't Broken > > It is rare to find a bug in the OS or the compiler, or even a third-party > product or library. The bug is most likely in the application.
  - AshamedCaptain 3y ago
    I really don't know when this became "common knowledge". Most OSes and compilers were utter crap not more than 10-20 years ago. We would spend afternoons ranting about MSVC++ bugs. gcc was a joke, and even if quickly improved, does nobody really remember that by the 2010's gcc was so buggy -O3 was barely usable? No Gentoo users around?
    - RetroTechie 3y ago
      I recall doing some Gentoo installs using -Os at least for some packages, just to see how much that would shrink binary sizes. Documentation / wikis etc. advised against that not just for performance-of-generated-code reasons, but also because of compiler bugs using -Os was considered unreliable. Note this was with ancient GCC (v2.9.x, early 3.x releases iirc)
    - paulddraper 3y ago
      `-O3` was known/documented to be unsafe.
  - toast0 3y ago
    Yeah, I don't know where that comes from. Everything is broken. If you think select is broken, read the docs and make sure you're using it correctly. If you still think select isn't doing what the docs say, try to reduce your actual use into a minimal test case and see. Go look at the source for select on your OS. Search the internet and see if anyone else ran into the same thing (but be careful, because there's lots of bad leads out there).
alphanullmeric 3y ago
[flagged]
- ComputerGuru 3y ago
  This has absolutely nothing to do with the software ecosystem or marketing decisions. The Samsung microcontroller/CPU development team is from a completely separate division (Samsung LSI) and makes these CPUs for use both by (the other parts of) Samsung and outside vendors.
  - perryizgr8 3y ago
    And every CPU in existence has errata. It's not like Samsung CPUs are any more filled with bugs than Qualcomm's.
jakub_g 3y ago
A few years ago I was doing Android native development. Back then Samsung as well as Huawei (and other cheap Chinese vendors) have been known to be doing weird things to the operating system and in Google Play console there'd often be wild device-specific crashes coming from those vendored Androids. I never loved Samsung devices but since then I avoid them even more.
satiric 3y ago
This site has a list of phones with the same CPU as the Samsung Galaxy S20 they mentioned. https://en.wikichip.org/wiki/qualcomm/snapdragon_800/865 https://en.wikichip.org/wiki/qualcomm/snapdragon_800/865 I wonder if it happens to them too?
- SketchySeaBeast 3y ago
  Given that it's only effecting S20's I'd be surprised if it was the snapdragon 865 as that's in a ton of phones as you point out. I'd expect it to be the Exynos 990, which is Samsung's custom chip for every region but North America and was only used in the S20 series.
  - scrlk 3y ago
    Wouldn't be surprised if it was an Exynos bug either. Samsung LSI haven't produced a decent flagship Exynos SOC for multiple generations now. Always behind Snapdragon on overall balance of performance (CPU/GPU/radios), heat and battery life. I refuse to buy any phone with an Exynos SOC ever since I got screwed with a poorly tuned Exynos 9810 in my S9. Andrei Frumusanu did a great two part article on turning the 9810 when he used to write for Anandtech: https://www.anandtech.com/show/12615/improving-exynos-9810-galaxy-s9-part-1 https://www.anandtech.com/show/12615/improving-exynos-9810-g... https://www.anandtech.com/show/12620/improving-the-exynos-9810-galaxy-s9-part-2 https://www.anandtech.com/show/12620/improving-the-exynos-98...
    - fifteen1506 3y ago
      Meh, decisions decisions. I bought a Pixel 6a which uses Samsung technology for the Google SoC. I bought it anyway because I wanted to install GrapheneOS but since I assumed it was evolved-but-still-Exynos I did hesitate.
    - AshamedCaptain 3y ago
      There used to be a reason: Exynos variants had unlockable bootloader. Not sure how feasible unlocking is nowadays on the Qualcomm variants.
dotdi 3y ago
A few years ago I also ran into a hardware (or at least very low-level software) bug that was only affecting Samsung devices. I had taken over maintenance of a library that allowed predicting satellite orbits from TLE data, which was used by the European Space Agency (and others) for the occasional mobile app. Sporadically, we were getting reports of strange situations where the altitude of a satellite was way off, by hundreds or thousands of kilometers. The bug was really difficult to track down and reproduce, and by chance one of our Samsung tablets finally started showing the behaviour. It turned out that Dalvik (the now obsolete Android VM) was figuring out that the TLE calculation was in a hot path and would JIT it to run more efficiently, and the optimized code used some native arithmetic (or was it trigonometic? can't recall) call that had a bug. We saw that the incorrect values started showing up right around the time when the JIT started kicking in. Fortunately, we could turn off JIT for specific devices and we rolled out an update that fixed the issue. Fun times!
- spuz 3y ago
  How do you trace what the JIT is doing in an Android app?
  - dotdi 3y ago
    If I recall correctly, there were some TRACE or DEBUG logs one could turn on that would - amid a ton of unhelpful output - also emit logs when the JIT was doing its thing. Then, while running a small reproducer app and combing through the output, we noticed values were always correct before the JIT ran, then when the JIT started to kick in, we had a transition period where sometimes wrong values started to crop up (I assume until all threads had switched to the optimized code), and then always produce the wrong result. This gave me a hunch that it could be related to JIT, and then I tried completely turning it off, which caused the issue to disappear.
FartyMcFarter 3y ago
It's never a compiler or CPU bug, until it is.
omoikane 3y ago
Summary: Samsung's LDRB doesn't zero-extend when reading a byte, so it's not following ARM specs[1]. Fix was to workaround in the JIT[2]. [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1833315#c13 https://bugzilla.mozilla.org/show_bug.cgi?id=1833315#c13 [2] https://hg.mozilla.org/mozilla-central/rev/c6a272593c07#l7.12 https://hg.mozilla.org/mozilla-central/rev/c6a272593c07#l7.1...
bch 3y ago
Here’s the first story I read (~11 years ago) with an occurrence like this that cracked my concept of hardware/software divide… https://www.theregister.com/2012/03/07/amd_opteron_bug_dragonfly/ https://www.theregister.com/2012/03/07/amd_opteron_bug_drago...
febed 3y ago
Kudos to these guys for figuring this out from the dump, not a trivial task! These kind of issues can take months to triage if one is unlucky. I remember our team going crazy trying to figure out the reason for random crashes until finally discovering a firmware bug in proprietary hardware.

246 points by kens 3y ago

LispSporks22 3y ago

The only thing better than being able to blame the compiler for something, is being able to blame the CPU

print_goto_ten 3y ago
Great tshirt
peheje 3y ago
Saving that quote.
amelius 3y ago
But how are you going to fix it then?
- wongarsu 3y ago
  Change your code to not trigger the bug, and leave that workaround there for the next decade or so until the buggy CPU generation isn't relevant anymore. Or hope for microcode updates to fix the bug at the CPU level.
  - amelius 3y ago
    That's my point ... it's a lot harder to do this than to just fix it in your own code.
- JonChesterfield 3y ago
  You change the compiler to carefully step around the stuff that doesn't work in the hardware.
- pajko 3y ago
  Compilers can be patched to automatically do some masking in the code generation step after every LDRB instruction in this case, when compiling for that specific CPU. For example, the Blackfin DSPs have many similar bugs which are fixed this way. Some of which the fixes done automatically, while others could be disabled for performance critical code which has been implemented while keeping the various issues in mind, like -mcsync-anomaly and -mspecld-anomaly. Certain MIPS CPUs also have incorrectly implemented instructions and the -mfix-* compiler parameters could be used to enable workarounds. JIT compilers can do the same.

jeffbee 3y ago

This is where the vocal minority of HN posters who are in a snit about telemetry are wrong. This is the kind of issue you can discover at global scale, but only if you collect the data.

n4r9 3y ago
It looks like this could have been discovered by simply having (anonymised) crash report telemetry? I don't know about others, but that sort of telemetry doesn't sound so bad.
- Nextgrid 3y ago
  It's very hard to have useful anonymized telemetry. In this case, you actually need at least a partial memory or register dump to understand what's going on, but those can contain sensitive data that the failing code was processing. Telemetry doesn't have to be 100% anonymized - it can (and in some cases needs) to contain PII, just needs to be collected respectfully and transparently. Ask the user before sending something, and let the user review it. The user can then make their own decision.
  - bpye 3y ago
    > Ask the user before sending something, and let the user review it. The user can then make their own decision. I under The sentiment, but say you were sending a memory dump as part of your crash reporting. How is a user going to review that?
mixedbit 3y ago
Such crash data can be collected with an old fashion 'Firefox has crashed, send a problem report to Mozilla [OK] [Cancel]' dialog, which is much less privacy-intrusive than telemetry.
- ksenzee 3y ago
  Right, and Firefox for Android does in fact collect its crash data like this.

garganzol 3y ago

When I was a kid, I always suspected a compiler or CPU bug when my program did not behave as expected. I heard similar stories on more than several occasions from other people, so being naïve must be a common thing among us software engineers when we make our first steps. Kudos to Mozilla, their finding is a real CPU bug.

pengaru 3y ago
For ages in *nixland the phrase "select is not broken" has been an aphorism for such situations. I'm not sure of its origin, but it's referenced in The Pragmatic Programmer: > Tip #33, pg. 95: > > “select” Isn't Broken > > It is rare to find a bug in the OS or the compiler, or even a third-party > product or library. The bug is most likely in the application.
- AshamedCaptain 3y ago
  I really don't know when this became "common knowledge". Most OSes and compilers were utter crap not more than 10-20 years ago. We would spend afternoons ranting about MSVC++ bugs. gcc was a joke, and even if quickly improved, does nobody really remember that by the 2010's gcc was so buggy -O3 was barely usable? No Gentoo users around?
  - RetroTechie 3y ago
    I recall doing some Gentoo installs using -Os at least for some packages, just to see how much that would shrink binary sizes. Documentation / wikis etc. advised against that not just for performance-of-generated-code reasons, but also because of compiler bugs using -Os was considered unreliable. Note this was with ancient GCC (v2.9.x, early 3.x releases iirc)
  - paulddraper 3y ago
    `-O3` was known/documented to be unsafe.
- toast0 3y ago
  Yeah, I don't know where that comes from. Everything is broken. If you think select is broken, read the docs and make sure you're using it correctly. If you still think select isn't doing what the docs say, try to reduce your actual use into a minimal test case and see. Go look at the source for select on your OS. Search the internet and see if anyone else ran into the same thing (but be careful, because there's lots of bad leads out there).

alphanullmeric 3y ago

[flagged]

ComputerGuru 3y ago
This has absolutely nothing to do with the software ecosystem or marketing decisions. The Samsung microcontroller/CPU development team is from a completely separate division (Samsung LSI) and makes these CPUs for use both by (the other parts of) Samsung and outside vendors.
- perryizgr8 3y ago
  And every CPU in existence has errata. It's not like Samsung CPUs are any more filled with bugs than Qualcomm's.

jakub_g 3y ago

A few years ago I was doing Android native development. Back then Samsung as well as Huawei (and other cheap Chinese vendors) have been known to be doing weird things to the operating system and in Google Play console there'd often be wild device-specific crashes coming from those vendored Androids. I never loved Samsung devices but since then I avoid them even more.

satiric 3y ago

This site has a list of phones with the same CPU as the Samsung Galaxy S20 they mentioned. https://en.wikichip.org/wiki/qualcomm/snapdragon_800/865 https://en.wikichip.org/wiki/qualcomm/snapdragon_800/865 I wonder if it happens to them too?

SketchySeaBeast 3y ago
Given that it's only effecting S20's I'd be surprised if it was the snapdragon 865 as that's in a ton of phones as you point out. I'd expect it to be the Exynos 990, which is Samsung's custom chip for every region but North America and was only used in the S20 series.
- scrlk 3y ago
  Wouldn't be surprised if it was an Exynos bug either. Samsung LSI haven't produced a decent flagship Exynos SOC for multiple generations now. Always behind Snapdragon on overall balance of performance (CPU/GPU/radios), heat and battery life. I refuse to buy any phone with an Exynos SOC ever since I got screwed with a poorly tuned Exynos 9810 in my S9. Andrei Frumusanu did a great two part article on turning the 9810 when he used to write for Anandtech: https://www.anandtech.com/show/12615/improving-exynos-9810-galaxy-s9-part-1 https://www.anandtech.com/show/12615/improving-exynos-9810-g... https://www.anandtech.com/show/12620/improving-the-exynos-9810-galaxy-s9-part-2 https://www.anandtech.com/show/12620/improving-the-exynos-98...
  - fifteen1506 3y ago
    Meh, decisions decisions. I bought a Pixel 6a which uses Samsung technology for the Google SoC. I bought it anyway because I wanted to install GrapheneOS but since I assumed it was evolved-but-still-Exynos I did hesitate.
  - AshamedCaptain 3y ago
    There used to be a reason: Exynos variants had unlockable bootloader. Not sure how feasible unlocking is nowadays on the Qualcomm variants.

dotdi 3y ago

A few years ago I also ran into a hardware (or at least very low-level software) bug that was only affecting Samsung devices. I had taken over maintenance of a library that allowed predicting satellite orbits from TLE data, which was used by the European Space Agency (and others) for the occasional mobile app. Sporadically, we were getting reports of strange situations where the altitude of a satellite was way off, by hundreds or thousands of kilometers. The bug was really difficult to track down and reproduce, and by chance one of our Samsung tablets finally started showing the behaviour. It turned out that Dalvik (the now obsolete Android VM) was figuring out that the TLE calculation was in a hot path and would JIT it to run more efficiently, and the optimized code used some native arithmetic (or was it trigonometic? can't recall) call that had a bug. We saw that the incorrect values started showing up right around the time when the JIT started kicking in. Fortunately, we could turn off JIT for specific devices and we rolled out an update that fixed the issue. Fun times!

spuz 3y ago
How do you trace what the JIT is doing in an Android app?
- dotdi 3y ago
  If I recall correctly, there were some TRACE or DEBUG logs one could turn on that would - amid a ton of unhelpful output - also emit logs when the JIT was doing its thing. Then, while running a small reproducer app and combing through the output, we noticed values were always correct before the JIT ran, then when the JIT started to kick in, we had a transition period where sometimes wrong values started to crop up (I assume until all threads had switched to the optimized code), and then always produce the wrong result. This gave me a hunch that it could be related to JIT, and then I tried completely turning it off, which caused the issue to disappear.

FartyMcFarter 3y ago

It's never a compiler or CPU bug, until it is.

omoikane 3y ago

Summary: Samsung's LDRB doesn't zero-extend when reading a byte, so it's not following ARM specs[1]. Fix was to workaround in the JIT[2]. [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1833315#c13 https://bugzilla.mozilla.org/show_bug.cgi?id=1833315#c13 [2] https://hg.mozilla.org/mozilla-central/rev/c6a272593c07#l7.12 https://hg.mozilla.org/mozilla-central/rev/c6a272593c07#l7.1...

bch 3y ago

Here’s the first story I read (~11 years ago) with an occurrence like this that cracked my concept of hardware/software divide… https://www.theregister.com/2012/03/07/amd_opteron_bug_dragonfly/ https://www.theregister.com/2012/03/07/amd_opteron_bug_drago...

febed 3y ago

Kudos to these guys for figuring this out from the dump, not a trivial task! These kind of issues can take months to triage if one is unlucky. I remember our team going crazy trying to figure out the reason for random crashes until finally discovering a firmware bug in proprietary hardware.