5 ms·

Someone's been messing with Python's floating point subnormals

209 points by fanf2 2y ago

bjornsing 2y ago
Is there a simple way to check if my Python script is affected? Because I guess numpy only complains if the FPU has been screwed up before it loads (not if some other package loaded later does it)?
gary_0 2y ago
(2022) I remember this one because of this part: > Unbeknownst to me, even with --dry-run pip will execute arbitrary code found in the package's setup.py. In fact, merely asking pip to download a package can execute arbitrary code (see pip issues 7325 and 1884 for more details)! So when I tried to dry-run install almost 400K Python packages, hilarity ensued. I spent a long time cleaning up the mess, and discovered some pretty poor setup.py practices along the way. But hey, at least I got two free pictures of anime catgirls, deposited directly into my home directory. Convenient!
- danwills 2y ago
  I don't use pip but tbh this makes me much less likely to even try it. So many things about it sound unsafe. Like dry-run not even working as you'd expect? I wonder if that got fixed after this article?
  - cozzyd 2y ago
    Probably the fix will be to rename the option --dank-run
- odyssey7 2y ago
  Why is this ecosystem considered acceptable? It’s because academia is a sort of anchor that preserves poor practices.
  - noahssnark 2y ago
    You'll have to ask every developer why it's acceptable, because they keep using this ecosystem. And then ask their managers. But this is what happens when you don't use package ecosystems with maintainers being run by adults.
    - physicsguy 2y ago
      If I wanted to compile a Go binary using an existing dynamically loaded C library, compiled with -ffast-math, what happens?
  - cozzyd 2y ago
    Well it's an ecosystem that is meant to be easy to use by people who don't know what they're doing.
  - Doxin 2y ago
    > Why is this ecosystem considered acceptable? It largely isn't, but python is a big ship and it takes time to turn. There's been a lot of movement in python packaging semi-recently and as far as I can tell using setup.py has been considered legacy for a while now. Biggest problem is that the new way to do packaging is not documented very well. It's split over three or four different projects. It mostly amounts to "you need to make a pyproject.toml file" but it's somewhat tricky to find what such a file is even supposed to contain.
    - morkalork 2y ago
      Again, why is this acceptable?
      - Joker_vD 2y ago
        Okay, it's unacceptable. Now what? Sadly, we (you, I, the Python leadership) can't just visit every single Python package publisher/maintainer and ask politely but firmly ask them to fix their packaging issues. And even if we could, at least half of them would either ignore us or would mess up in some new, exciting ways while trying to adapt to the flavour du jour of Python packaging.
- poikroequ 2y ago
  I've had to clean up after npm packages as well. Git hooks I didn't ask for. Shell modifications I didn't ask for. I hope that, one day, these package managers will utilize sandboxing/containerization to avoid messes like this.
  - richardfey 2y ago
    Python and node software is something I always try to run in containers for convenience and security reasons
- zbentley 2y ago
  The "--only-binary=:all:" option should force Pip to only install packages that provide wheels, which doesn't run arbitrary setup.py code (right? I've been assuming it doesn't and really hope that is a valid assumption). That will cause some installations to fail if wheels are not available. However, given that wheels are increasingly common (even for pure-python "source" packages), this can be used as a sort of bisect to enumerate and isolate/audit/file issues on/sandbox/etc. the remaining setup.py-based distributions.
  - physicsguy 2y ago
    Depends on the platform you're on, some platforms have very few wheels.
- Sakos 2y ago
  Naive question, does this have the same effect if I use virtual environments?
  - plorkyeran 2y ago
    Yes. Virtual environments aren’t a full sandbox.
  - aidenn0 2y ago
    Yes, because the line that does it is literally: os.system('cp -r ./pics/ ~/.local/share/koneko/')
    - cozzyd 2y ago
      that's a feature. You can conveniently do pip install nukemydrive #no, this package doesn't exist (yet) rather than than the harder to remember rm -rf --no-preserve-root /
- aidenn0 2y ago
  Here's the line that copies those pictures in: https://github.com/akazukin5151/koneko/blob/master/setup.py#L18 https://github.com/akazukin5151/koneko/blob/master/setup.py#...
Arech 2y ago
That's very important info for some pythonistas, thanks for sharing!
spacebacon 2y ago
2001 - GCC 3.0: - Introduction of `-ffast-math`: The `-ffast-math` flag was introduced, enabling aggressive floating-point optimizations at the cost of strict IEEE compliance. 2004 - GCC 3.4: - Refinements and Extensions: Additional optimizations added, including better handling of subnormal numbers and more aggressive operation reordering. 2007 - GCC 4.2: - Introduction of Related Flags: `-funsafe-math-optimizations` flag introduced, offering more granular control over specific optimizations. 2010 - GCC 4.5: - Improvements in Vectorization: Enhanced vectorization capabilities, particularly for SIMD hardware using SSE and AVX instruction sets. 2013 - GCC 4.8: - More Granular Control: Introduction of flags like `-fno-math-errno`, improving efficiency by assuming mathematical functions do not set `errno`. 2017 - GCC 7.0: - Enhanced Complex Number Optimizations: Improved performance for complex number arithmetic, benefiting scientific and engineering applications. 2021 - GCC 11.0: - Better Support for Modern Hardware: Optimizations leveraging modern CPU architectures and instruction sets like AVX-512. 2024 - GCC 13.0 (Experimental): - Experimental Features: Additional optimizations focused on new CPU features and better handling of edge cases. Sources: - GCC documentation archives - Release notes from various GCC versions - [GCC Wiki](https://gcc.gnu.org/wiki/ https://gcc.gnu.org/wiki/) - [Krister Walfridsson's blog](https://kristerw.github.io https://kristerw.github.io)
- amluto 2y ago
  Your timeline is missing: 2012: I file the obvious bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55522 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55522 Early 2023: bug is fixed. Feb 2024: clang follows suit: https://github.com/llvm/llvm-project/pull/80475 https://github.com/llvm/llvm-project/pull/80475 And maybe later this year this problem will finally be gone in common Linux distros.
  - SAI_Peregrinus 2y ago
    Plenty of OSS authors are contributing from work, so there's a good chance they're running LTS distros. Give it another few years for those to get updated.
  - duped 2y ago
    I don't want to read through 11 years of comments, but what does observable effects does corrupting mxcsr have? Surely it's just enabling FTZ/DAZ mode for the FPU which could harm bit-exactness in floating point math... but in code that cares about it, it's not uncommon to see RAII guards to configure the behavior you want (and usually, it's desirable to not comply with IEEE754 w.r.t subnormal numbers). Like it's a quirk that floating point math can be non-deterministic but how bad is it, and is it actually a bug?
mguijarr 2y ago
Contrary to what is said in the article, gevent has the fix, it has been merged as https://github.com/gevent/gevent/commit/e29bd2ee11ca5f78cc9c9bc784d875681afb8d23 https://github.com/gevent/gevent/commit/e29bd2ee11ca5f78cc9c... 2 years ago.
- cbsks 2y ago
  That fix was a month after this article was written.
pimlottc 2y ago
What's the real-world consequences of having the floating point behavior changed? The article mentions some types of iterative algorithms but it's not clear how often those would be used. Would be interested to know what actual issues arose in any downstream projects.
- adrian_b 2y ago
  Computation errors that will be discovered only with great difficulties, when someone is careful enough to use some additional verification methods for the numeric results. The magnitude of the error caused by an underflow is completely unpredictable, which is why in any serious program it is unacceptable to ignore the underflows. Even when the errors happen to be discovered soon enough, people will be puzzled about their origin and they may waste a lot of time analyzing their algorithms, because there are a lot of things that can cause numeric errors. Because at their origin the underflows will neither signal any exception nor generate any unusual value, the errors will be normally caught much later, perhaps after thousands of other operations, when their cause will not be obvious. This fight between the people who want to get only correct results from computers and the people who do not care whether the results are erroneous as long as the results are obtained after a delay shorter by a few percents than for obtaining correct results has continued for decades, almost since electronic computers have been invented. Errors are acceptable in games or for some other graphics or audio applications that generate ephemeral images or sounds, but they are not acceptable for any engineering purposes. The option "-Ofast" is like smoking. There is no doubt that it is a bad habit, but whoever wants to smoke should be free to do it. On the other hand, exactly like a smoker should not be permitted to smoke in a closed room with non-smokers, the option "-Ofast" and the related options should not be permitted to alter the behavior of any other programs that are linked with an object file compiled with it. The behavior of "-Ofast" where it affects globally the content of MXCSR is unacceptable. The right behavior would have been for any function compiled with "-Ofast" to save MXCSR, put in it any desired value for the duration of the function execution and restore it at function exit. Moreover, when invoking any other function it should restore the original MXCSR before the call and put again in it the desired local value after the function returns back.
  - 2y ago
reisse 2y ago
As always, -funsafe-math-optimizations are neither fun nor safe
- cwillu 2y ago
  It's +funsafe for those.
  - iforgotpassword 2y ago
    -movflags +funsafe
- tgsovlerkhgsel 2y ago
  I find it interesting that -ffast-math includes -funsafe-math-optimizations. Having an option that says "unsafe" creates the expectation that unsafe options are labeled like that, and then enabling said unsafe option from another option that doesn't have "unsafe" in the name is a bit surprising.
sulam 2y ago
<dang>, this is (2022).
SillyUsername 2y ago
Well that explains the LLMs getting their answers wrong ;)
dhosek 2y ago
The bit about the behavior propagating through shared libraries is yet another reason to prefer static linking.
- fanf2 2y ago
  It propagates through static linking too, if anything in the program is compiled with -Ofast
  - JonChesterfield 2y ago
    It might or might not depending on whether the compiler leaves corresponding state in the intermediate files. This might toggle depending on -fLTO. Ofast is not a terribly good idea really.
csours 2y ago
Kind of related topic: Using libraries is a pain - Is there any evidence that people are using LLM coding tools to write library functions instead of importing libraries? How would we tell? Can you think of second order effects from this?
- cratermoon 2y ago
  The LLM coding tools will likely reproduce the errors from the code they were trained on, including compiler options like -Ofast. Because they tools are generating text based on tokens, the -Ofast option can and likely will appear in contexts where it is completely inappropriate. Programmers who use this code will propagate errors found and fixed 2 years ago.
  - csours 2y ago
    > propagate errors found and fixed 2 years ago Ahh! Yeah, we need better value functions. https://news.ycombinator.com/item?id=41188647 https://news.ycombinator.com/item?id=41188647
  - mhh__ 2y ago
    I'd take a bet against that actually as people about fastmath is exactly the kind of hand holding OpenAI in particular train for.
JSDevOps 2y ago
I just wanted to say—this is an absolutely brilliant write-up. Thanks so much for all your hard work
joshlk 2y ago
> I have never met a scientist who can resist the lure of fast-but-dangerous math This made me chuckle
jhj 2y ago
The original sin here is that original 1980s designs carry over: the processor retains FP unit state, rather than each instruction indicating what subnormal flush mode (or rounding mode or whatever) one wishes to use with no retained FP unit state. See also: the IEEE FP exception design (e.g., signaling NaNs) causing havoc with SIMD, deep pipelining, out-of-order execution etc.
throwaway81523 2y ago
This is worth reading all the way through. Ouch.
im3w1l 2y ago
After thinking about this a bit, my conclusion is that it's a deficiency in the calling convention. It should specify the floating point flags as either caller saved or callee saved.

209 points by fanf2 2y ago

Is there a simple way to check if my Python script is affected? Because I guess numpy only complains if the FPU has been screwed up before it loads (not if some other package loaded later does it)?

gary_0 2y ago

(2022) I remember this one because of this part: > Unbeknownst to me, even with --dry-run pip will execute arbitrary code found in the package's setup.py. In fact, merely asking pip to download a package can execute arbitrary code (see pip issues 7325 and 1884 for more details)! So when I tried to dry-run install almost 400K Python packages, hilarity ensued. I spent a long time cleaning up the mess, and discovered some pretty poor setup.py practices along the way. But hey, at least I got two free pictures of anime catgirls, deposited directly into my home directory. Convenient!

danwills 2y ago
I don't use pip but tbh this makes me much less likely to even try it. So many things about it sound unsafe. Like dry-run not even working as you'd expect? I wonder if that got fixed after this article?
- cozzyd 2y ago
  Probably the fix will be to rename the option --dank-run
odyssey7 2y ago
Why is this ecosystem considered acceptable? It’s because academia is a sort of anchor that preserves poor practices.
- noahssnark 2y ago
  You'll have to ask every developer why it's acceptable, because they keep using this ecosystem. And then ask their managers. But this is what happens when you don't use package ecosystems with maintainers being run by adults.
  - physicsguy 2y ago
    If I wanted to compile a Go binary using an existing dynamically loaded C library, compiled with -ffast-math, what happens?
- cozzyd 2y ago
  Well it's an ecosystem that is meant to be easy to use by people who don't know what they're doing.
- Doxin 2y ago
  > Why is this ecosystem considered acceptable? It largely isn't, but python is a big ship and it takes time to turn. There's been a lot of movement in python packaging semi-recently and as far as I can tell using setup.py has been considered legacy for a while now. Biggest problem is that the new way to do packaging is not documented very well. It's split over three or four different projects. It mostly amounts to "you need to make a pyproject.toml file" but it's somewhat tricky to find what such a file is even supposed to contain.
  - morkalork 2y ago
    Again, why is this acceptable?
    - Joker_vD 2y ago
      Okay, it's unacceptable. Now what? Sadly, we (you, I, the Python leadership) can't just visit every single Python package publisher/maintainer and ask politely but firmly ask them to fix their packaging issues. And even if we could, at least half of them would either ignore us or would mess up in some new, exciting ways while trying to adapt to the flavour du jour of Python packaging.
poikroequ 2y ago
I've had to clean up after npm packages as well. Git hooks I didn't ask for. Shell modifications I didn't ask for. I hope that, one day, these package managers will utilize sandboxing/containerization to avoid messes like this.
- richardfey 2y ago
  Python and node software is something I always try to run in containers for convenience and security reasons
zbentley 2y ago
The "--only-binary=:all:" option should force Pip to only install packages that provide wheels, which doesn't run arbitrary setup.py code (right? I've been assuming it doesn't and really hope that is a valid assumption). That will cause some installations to fail if wheels are not available. However, given that wheels are increasingly common (even for pure-python "source" packages), this can be used as a sort of bisect to enumerate and isolate/audit/file issues on/sandbox/etc. the remaining setup.py-based distributions.
- physicsguy 2y ago
  Depends on the platform you're on, some platforms have very few wheels.
Sakos 2y ago
Naive question, does this have the same effect if I use virtual environments?
- plorkyeran 2y ago
  Yes. Virtual environments aren’t a full sandbox.
- aidenn0 2y ago
  Yes, because the line that does it is literally: os.system('cp -r ./pics/ ~/.local/share/koneko/')
  - cozzyd 2y ago
    that's a feature. You can conveniently do pip install nukemydrive #no, this package doesn't exist (yet) rather than than the harder to remember rm -rf --no-preserve-root /
aidenn0 2y ago
Here's the line that copies those pictures in: https://github.com/akazukin5151/koneko/blob/master/setup.py#L18 https://github.com/akazukin5151/koneko/blob/master/setup.py#...

Arech 2y ago

That's very important info for some pythonistas, thanks for sharing!

spacebacon 2y ago

2001 - GCC 3.0: - Introduction of `-ffast-math`: The `-ffast-math` flag was introduced, enabling aggressive floating-point optimizations at the cost of strict IEEE compliance. 2004 - GCC 3.4: - Refinements and Extensions: Additional optimizations added, including better handling of subnormal numbers and more aggressive operation reordering. 2007 - GCC 4.2: - Introduction of Related Flags: `-funsafe-math-optimizations` flag introduced, offering more granular control over specific optimizations. 2010 - GCC 4.5: - Improvements in Vectorization: Enhanced vectorization capabilities, particularly for SIMD hardware using SSE and AVX instruction sets. 2013 - GCC 4.8: - More Granular Control: Introduction of flags like `-fno-math-errno`, improving efficiency by assuming mathematical functions do not set `errno`. 2017 - GCC 7.0: - Enhanced Complex Number Optimizations: Improved performance for complex number arithmetic, benefiting scientific and engineering applications. 2021 - GCC 11.0: - Better Support for Modern Hardware: Optimizations leveraging modern CPU architectures and instruction sets like AVX-512. 2024 - GCC 13.0 (Experimental): - Experimental Features: Additional optimizations focused on new CPU features and better handling of edge cases. Sources: - GCC documentation archives - Release notes from various GCC versions - [GCC Wiki](https://gcc.gnu.org/wiki/ https://gcc.gnu.org/wiki/) - [Krister Walfridsson's blog](https://kristerw.github.io https://kristerw.github.io)

amluto 2y ago
Your timeline is missing: 2012: I file the obvious bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55522 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55522 Early 2023: bug is fixed. Feb 2024: clang follows suit: https://github.com/llvm/llvm-project/pull/80475 https://github.com/llvm/llvm-project/pull/80475 And maybe later this year this problem will finally be gone in common Linux distros.
- SAI_Peregrinus 2y ago
  Plenty of OSS authors are contributing from work, so there's a good chance they're running LTS distros. Give it another few years for those to get updated.
- duped 2y ago
  I don't want to read through 11 years of comments, but what does observable effects does corrupting mxcsr have? Surely it's just enabling FTZ/DAZ mode for the FPU which could harm bit-exactness in floating point math... but in code that cares about it, it's not uncommon to see RAII guards to configure the behavior you want (and usually, it's desirable to not comply with IEEE754 w.r.t subnormal numbers). Like it's a quirk that floating point math can be non-deterministic but how bad is it, and is it actually a bug?

mguijarr 2y ago

Contrary to what is said in the article, gevent has the fix, it has been merged as https://github.com/gevent/gevent/commit/e29bd2ee11ca5f78cc9c9bc784d875681afb8d23 https://github.com/gevent/gevent/commit/e29bd2ee11ca5f78cc9c... 2 years ago.

cbsks 2y ago
That fix was a month after this article was written.

pimlottc 2y ago

What's the real-world consequences of having the floating point behavior changed? The article mentions some types of iterative algorithms but it's not clear how often those would be used. Would be interested to know what actual issues arose in any downstream projects.

adrian_b 2y ago
Computation errors that will be discovered only with great difficulties, when someone is careful enough to use some additional verification methods for the numeric results. The magnitude of the error caused by an underflow is completely unpredictable, which is why in any serious program it is unacceptable to ignore the underflows. Even when the errors happen to be discovered soon enough, people will be puzzled about their origin and they may waste a lot of time analyzing their algorithms, because there are a lot of things that can cause numeric errors. Because at their origin the underflows will neither signal any exception nor generate any unusual value, the errors will be normally caught much later, perhaps after thousands of other operations, when their cause will not be obvious. This fight between the people who want to get only correct results from computers and the people who do not care whether the results are erroneous as long as the results are obtained after a delay shorter by a few percents than for obtaining correct results has continued for decades, almost since electronic computers have been invented. Errors are acceptable in games or for some other graphics or audio applications that generate ephemeral images or sounds, but they are not acceptable for any engineering purposes. The option "-Ofast" is like smoking. There is no doubt that it is a bad habit, but whoever wants to smoke should be free to do it. On the other hand, exactly like a smoker should not be permitted to smoke in a closed room with non-smokers, the option "-Ofast" and the related options should not be permitted to alter the behavior of any other programs that are linked with an object file compiled with it. The behavior of "-Ofast" where it affects globally the content of MXCSR is unacceptable. The right behavior would have been for any function compiled with "-Ofast" to save MXCSR, put in it any desired value for the duration of the function execution and restore it at function exit. Moreover, when invoking any other function it should restore the original MXCSR before the call and put again in it the desired local value after the function returns back.
- 2y ago

reisse 2y ago

As always, -funsafe-math-optimizations are neither fun nor safe

cwillu 2y ago
It's +funsafe for those.
- iforgotpassword 2y ago
  -movflags +funsafe
tgsovlerkhgsel 2y ago
I find it interesting that -ffast-math includes -funsafe-math-optimizations. Having an option that says "unsafe" creates the expectation that unsafe options are labeled like that, and then enabling said unsafe option from another option that doesn't have "unsafe" in the name is a bit surprising.

sulam 2y ago

<dang>, this is (2022).

SillyUsername 2y ago

Well that explains the LLMs getting their answers wrong ;)

dhosek 2y ago

The bit about the behavior propagating through shared libraries is yet another reason to prefer static linking.

fanf2 2y ago
It propagates through static linking too, if anything in the program is compiled with -Ofast
- JonChesterfield 2y ago
  It might or might not depending on whether the compiler leaves corresponding state in the intermediate files. This might toggle depending on -fLTO. Ofast is not a terribly good idea really.

csours 2y ago

Kind of related topic: Using libraries is a pain - Is there any evidence that people are using LLM coding tools to write library functions instead of importing libraries? How would we tell? Can you think of second order effects from this?

cratermoon 2y ago
The LLM coding tools will likely reproduce the errors from the code they were trained on, including compiler options like -Ofast. Because they tools are generating text based on tokens, the -Ofast option can and likely will appear in contexts where it is completely inappropriate. Programmers who use this code will propagate errors found and fixed 2 years ago.
- csours 2y ago
  > propagate errors found and fixed 2 years ago Ahh! Yeah, we need better value functions. https://news.ycombinator.com/item?id=41188647 https://news.ycombinator.com/item?id=41188647
- mhh__ 2y ago
  I'd take a bet against that actually as people about fastmath is exactly the kind of hand holding OpenAI in particular train for.

JSDevOps 2y ago

I just wanted to say—this is an absolutely brilliant write-up. Thanks so much for all your hard work

joshlk 2y ago

> I have never met a scientist who can resist the lure of fast-but-dangerous math This made me chuckle

jhj 2y ago

The original sin here is that original 1980s designs carry over: the processor retains FP unit state, rather than each instruction indicating what subnormal flush mode (or rounding mode or whatever) one wishes to use with no retained FP unit state. See also: the IEEE FP exception design (e.g., signaling NaNs) causing havoc with SIMD, deep pipelining, out-of-order execution etc.

throwaway81523 2y ago

This is worth reading all the way through. Ouch.

im3w1l 2y ago

After thinking about this a bit, my conclusion is that it's a deficiency in the calling convention. It should specify the floating point flags as either caller saved or callee saved.