6 ms·

Homebrew: AVX instructions in Big Sur bottles crash when run in Rosetta 2

54 points by thibautg 6y ago

marcan_42 6y ago
TL;DR: A change was made to set the minimum CPU level (instruction set target) for Big Sur builds to Ivy Bridge. This assumes the presence of AVX instructions on all Big Sur machines, which is true for Intel macs. Rosetta is a Big Sur "machine" that does not support AVX. Therefore, the assumption was false.
- phire 6y ago
  > which is true for Intel macs But not true for Intel in general, because Intel still ships low-end Core processes branded as "Pentium" and "Celeron" such as the Intel® Pentium® Gold G6600 which have AVX disabled. It only holds true for Intel macs because Apple have never used these processors in their products.
  - daenney 6y ago
    It doesn’t matter that it’s not true for Intel in general, brew doesn’t target “Intel in general”.
    - Brian_K_White 6y ago
      If it doesn't, that's just yet another broken assumption by homebrew. Macos is run on other hardware, both physical and virtual, than a few Apple models.
      - daenney 6y ago
        It’s not a broken assumption. You can still build and compile all formulae yourself. But for the bottles, the precompiled builds, they’ve always had a lower boundary on which generation of Mac you need to at least be running, and as a consequence which instructions are available.
adenylyl 6y ago
Why is this interesting? Rosetta 2 is documented to not support AVX: https://developer.apple.com/documentation/apple_silicon/about_the_rosetta_translation_environment https://developer.apple.com/documentation/apple_silicon/abou...
- thibautg 6y ago
  I own a MacBook Air M1. Today, I noticed that my Homebrew Python (3.9.1 on Intel) installation had reverted to the default one (3.8 on ARM). I forced a reinstall with homebrew and I got an ‘illegal hardware instruction’ error when running python3. It was corrected by building from source (with the -s switch: brew reinstall -s python3). I found out about this issue on Github and I learned that the M1 chip does not support AVX (which is indeed documented) and that Homebrew developers assumed that all CPUs supported by Big Sur were compatible with AVX instruction set (which is true except for the M1 under Rosetta 2). I found it interesting to see how they rapidly corrected the situation by merging a partial revert. I also found their CI infrastructure interesting to watch while it was live-testing the commit.
  - klodolph 6y ago
    This is fascinating. I can understand why the Homebrew devs might make the decision, but it seems the right way to do this is just to, e.g., cc -dM -E -xc /dev/null | less Anything with a preprocessor macro is supported on all OS configurations that you are targeting. You can see, for example, that __SSE4_1__ is defined, so SSE 4.1 is ok to use without checking it at runtime with e.g. sysctlbyname("hw.optional.sse4_1", &enabled, ...) I have always thought that it is worth a bit of paranoia to try and do the check for optional features the “correct” way, whatever method the OS advertises (like sysctl for macOS) rather than doing something like cpuid. After all, every once in a while it’s possible to run into a configuration where the CPU does support some particular feature but the OS does not, and if you rely on cpuid you could be up shit creek without a paddle.
    - bodyfour 6y ago
      It was a mistaken assumption from the Homebrew developers who didn't realize AVX would cause problems with Rosetta2. Within a few days problems became evident and Homebrew turned the compiler flag back off. Now it's just a question of the best way of remediating the already-compiled packages in the simplest way possible. Really the fact that this is now a front-page-of-HN story is going to cause more confusion to people. It was just a short-lived bug.
a-dub 6y ago
why doesn't rosetta support avx? arm has neon and sve... i would assume that there's equivalence for most basic vector math instructions and for those that there isn't, like the weird sha256 and aes instructions, they could just be emulated in software?
- daenney 6y ago
  Over here: https://news.ycombinator.com/item?id=25548001 https://news.ycombinator.com/item?id=25548001. Intel patents.
surfer7837 6y ago
Only bottles are affected so you can just compile from source with homebrew
mhh__ 6y ago
Is there any long-form documentation (specifically an optimization manual) for M1 yet?. Currently the only details I can find about the microarchitecture is either press waffle or things reverse engineered via timing (you can look for bumps in graphs to find the widths of various speculative features of the CPU). It would be very Apple not to publish any (Fucking sad if true) Edit: Intel, like them or not, have genuinely very good documentation (Much better than AMD's) and seem to have really thought about actually using their performance - AMD can really deliver the goods on their terms, but they still lag behind in both software (particular compared to Nvidia) and documentation (Intel's optimization manual is 850 pages; AMD's is 45 pages). AMD basically give you a raw list of RDPMC event assignments to play with, Intel actually tell you how to use them.
- skavi 6y ago
  If Apple wants to move fast, maybe it would be disadvantageous to have developers hyper optimize for a specific μarch?
  - mhh__ 6y ago
    No, document them both. And you can't really move fast, progress comes in a big dollop with sprinkles on top (i.e. Zen brought like 50% improvement over Excavator but the current AMD is Zen 3 rather than Ben 1)
- saagarjha 6y ago
  There is not.
- giomasce 6y ago
  Isn't the optimization manual for Rosetta basically "Recompile for ARM"?
  - mhh__ 6y ago
    And when you've done that what next?
- 6y ago
chrisseaton 6y ago
Why aren't they checking if the CPU supports AVX before using it?!
- lights0123 6y ago
  Because all Intel Macs that support Big Sur have AVX, so the Homebrew team enabled it globally. It allows for "things like small memset() operations [to get] inlined as AVX instructions" as the compiler knows that an AVX operation is faster than a function call.
- bodyfour 6y ago
  It was just a (short-lived) bug, not a conscious decision to break anything. Not sure why this got so much traction on HN.
recursive 6y ago
What's a Big Sur bottle? I don't follow Apple stuff super closely, but I'm completely lost here. Searches aren't too helpful.
- grzm 6y ago
  It's not an Apple thing; it's homebrew's binary packaging: https://docs.brew.sh/Bottles https://docs.brew.sh/Bottles
- ficklepickle 6y ago
  They use their own special snowflake jargon because they are Mac people. Best to not waste brain cycles on it.

54 points by thibautg 6y ago

TL;DR: A change was made to set the minimum CPU level (instruction set target) for Big Sur builds to Ivy Bridge. This assumes the presence of AVX instructions on all Big Sur machines, which is true for Intel macs. Rosetta is a Big Sur "machine" that does not support AVX. Therefore, the assumption was false.

phire 6y ago
> which is true for Intel macs But not true for Intel in general, because Intel still ships low-end Core processes branded as "Pentium" and "Celeron" such as the Intel® Pentium® Gold G6600 which have AVX disabled. It only holds true for Intel macs because Apple have never used these processors in their products.
- daenney 6y ago
  It doesn’t matter that it’s not true for Intel in general, brew doesn’t target “Intel in general”.
  - Brian_K_White 6y ago
    If it doesn't, that's just yet another broken assumption by homebrew. Macos is run on other hardware, both physical and virtual, than a few Apple models.
    - daenney 6y ago
      It’s not a broken assumption. You can still build and compile all formulae yourself. But for the bottles, the precompiled builds, they’ve always had a lower boundary on which generation of Mac you need to at least be running, and as a consequence which instructions are available.

adenylyl 6y ago

Why is this interesting? Rosetta 2 is documented to not support AVX: https://developer.apple.com/documentation/apple_silicon/about_the_rosetta_translation_environment https://developer.apple.com/documentation/apple_silicon/abou...

thibautg 6y ago
I own a MacBook Air M1. Today, I noticed that my Homebrew Python (3.9.1 on Intel) installation had reverted to the default one (3.8 on ARM). I forced a reinstall with homebrew and I got an ‘illegal hardware instruction’ error when running python3. It was corrected by building from source (with the -s switch: brew reinstall -s python3). I found out about this issue on Github and I learned that the M1 chip does not support AVX (which is indeed documented) and that Homebrew developers assumed that all CPUs supported by Big Sur were compatible with AVX instruction set (which is true except for the M1 under Rosetta 2). I found it interesting to see how they rapidly corrected the situation by merging a partial revert. I also found their CI infrastructure interesting to watch while it was live-testing the commit.
- klodolph 6y ago
  This is fascinating. I can understand why the Homebrew devs might make the decision, but it seems the right way to do this is just to, e.g., cc -dM -E -xc /dev/null | less Anything with a preprocessor macro is supported on all OS configurations that you are targeting. You can see, for example, that __SSE4_1__ is defined, so SSE 4.1 is ok to use without checking it at runtime with e.g. sysctlbyname("hw.optional.sse4_1", &enabled, ...) I have always thought that it is worth a bit of paranoia to try and do the check for optional features the “correct” way, whatever method the OS advertises (like sysctl for macOS) rather than doing something like cpuid. After all, every once in a while it’s possible to run into a configuration where the CPU does support some particular feature but the OS does not, and if you rely on cpuid you could be up shit creek without a paddle.
  - bodyfour 6y ago
    It was a mistaken assumption from the Homebrew developers who didn't realize AVX would cause problems with Rosetta2. Within a few days problems became evident and Homebrew turned the compiler flag back off. Now it's just a question of the best way of remediating the already-compiled packages in the simplest way possible. Really the fact that this is now a front-page-of-HN story is going to cause more confusion to people. It was just a short-lived bug.

a-dub 6y ago

why doesn't rosetta support avx? arm has neon and sve... i would assume that there's equivalence for most basic vector math instructions and for those that there isn't, like the weird sha256 and aes instructions, they could just be emulated in software?

daenney 6y ago
Over here: https://news.ycombinator.com/item?id=25548001 https://news.ycombinator.com/item?id=25548001. Intel patents.

surfer7837 6y ago

Only bottles are affected so you can just compile from source with homebrew

mhh__ 6y ago

Is there any long-form documentation (specifically an optimization manual) for M1 yet?. Currently the only details I can find about the microarchitecture is either press waffle or things reverse engineered via timing (you can look for bumps in graphs to find the widths of various speculative features of the CPU). It would be very Apple not to publish any (Fucking sad if true) Edit: Intel, like them or not, have genuinely very good documentation (Much better than AMD's) and seem to have really thought about actually using their performance - AMD can really deliver the goods on their terms, but they still lag behind in both software (particular compared to Nvidia) and documentation (Intel's optimization manual is 850 pages; AMD's is 45 pages). AMD basically give you a raw list of RDPMC event assignments to play with, Intel actually tell you how to use them.

skavi 6y ago
If Apple wants to move fast, maybe it would be disadvantageous to have developers hyper optimize for a specific μarch?
- mhh__ 6y ago
  No, document them both. And you can't really move fast, progress comes in a big dollop with sprinkles on top (i.e. Zen brought like 50% improvement over Excavator but the current AMD is Zen 3 rather than Ben 1)
saagarjha 6y ago
There is not.
giomasce 6y ago
Isn't the optimization manual for Rosetta basically "Recompile for ARM"?
- mhh__ 6y ago
  And when you've done that what next?
6y ago

chrisseaton 6y ago

Why aren't they checking if the CPU supports AVX before using it?!

lights0123 6y ago
Because all Intel Macs that support Big Sur have AVX, so the Homebrew team enabled it globally. It allows for "things like small memset() operations [to get] inlined as AVX instructions" as the compiler knows that an AVX operation is faster than a function call.
bodyfour 6y ago
It was just a (short-lived) bug, not a conscious decision to break anything. Not sure why this got so much traction on HN.

recursive 6y ago

What's a Big Sur bottle? I don't follow Apple stuff super closely, but I'm completely lost here. Searches aren't too helpful.

grzm 6y ago
It's not an Apple thing; it's homebrew's binary packaging: https://docs.brew.sh/Bottles https://docs.brew.sh/Bottles
ficklepickle 6y ago
They use their own special snowflake jargon because they are Mac people. Best to not waste brain cycles on it.