7 ms·

Why do you say almost never? Don’t let the name scare you; all floating point math is inaccurate. Fast math is only slightly less accurate, I think typically it

by dahart 3y ago

Why do you say almost never? Don’t let the name scare you; all floating point math is inaccurate. Fast math is only slightly less accurate, I think typically it’s a 1 or maybe 2 LSB difference. At least in CUDA it is, and I think many (most?) people & situations can tolerate 22 bits of mantissa compared to 23, and many (most?) people/situations aren’t paying attention to inf/nan/exception issues at all. I deal with a lot of floating point professionally day to day, and I use fast math all the time, since the tradeoff for higher performance and the relatively small loss of accuracy are acceptable. Maybe the biggest issue I run into is lack of denorms with CUDA fast-math, and it’s pretty rare for me to care about numbers smaller than 10^-38. Heck, I’d say I can tolerate 8 or 16 bits of mantissa most of the time, and fast-math floats are way more accurate than that. And we know a lot of neural network training these days can tolerate less than 8 bits of mantissa.

mort96 3y ago
The scary thing IMO is: your code might be fine with unsafe math optimisations, but maybe you're using a library which is written to do operations in a certain order to minimise numerical error, and unsafe math operations changes the code which are mathematically equivalent but which results in many orders of magnitude more numerical error. It's probably fine most of the time, but it's kinda scary.
- deleted 3y ago
  [deleted]
- dahart 3y ago
  It shouldn’t be scary. Any library that is sensitive to order of operations will hopefully have a big fat warning on it. And it can be compiled separately with fast-math disabled. I don’t know of any such libraries off the top of my head, and it’s quite rare to find situations that result in orders of magnitude more error, though I grant you it can happen, and it can be contrived pretty easily.
  - planede 3y ago
    You can't fully disable fast-math per-library, moreover a library compiled with fast-math might also introduce inaccuracies in a seemingly unrelated library or application code in the same executable. The reason is that fast-math enables some dynamic initialization of the library that changes the floating point environment in some ways.
    - londons_explore 3y ago
      you're gonna hive to give us a concrete real world example to convince most of us...
    - dahart 3y ago
      > You can’t fully disable fast-math per library Can you elaborate? What fast-math can sneak into a library that disabled fast-math at compile time? > fast-math enables some dynamic initialization of the library that changes the floating point environment in some ways. I wasn’t aware of this, I would love to see some documentation discussing exactly what happens, can you send a link?
      - mort96 3y ago
        > Can you elaborate? What fast-math can sneak into a library that disabled fast-math at compile time? A lot of library code is in headers (especially in C++!). The code in headers is compiled by your compiler using your compile options.
        dahart 3y ago
        Ah, of course, very good point. A header-only library doesn’t have separate compile options. This is a great reason for a float-sensitive library to not be header-only, right?
        mort96 3y ago
        It's not just about being header-only, lots of libraries which aren't header-only still have code in headers. The library may choose to put certain functions in headers for performance reasons (to let compiler inline them), or, in C++, function templates and class templates generally have to be in headers. But yeah, it's probably a good idea to not put code which breaks under -ffast-math in headers if possible.
usefulcat 3y ago
> Fast math is only slightly less accurate 'slightly'? Last I checked, -Ofast completely breaks std::isnan and std::isinf--they always return false.
- dahart 3y ago
  Hopefully it was clear from the rest of my comment that I was talking about in-range floats there. I wouldn’t necessarily call inf & nan handling an accuracy issue, that’s more about exceptional cases, but to your point I would have to agree that losing std::isinf is kinda bad since divide by zero is probably near the very top of the list of things most people using floats casually might have to deal with. Which compiler are you using where std::isinf breaks? Hopefully it was also clear that my experience leans toward CUDA, and I think the inf & nan support works there in the presence of NVCC’s fast-math.
  - usefulcat 3y ago
    My experience is with gcc and clang on x86. I generally agree with you regarding accuracy, which is why I was quite surprised when I first discovered that -Ofast breaks isnan/isinf. Even if I don't care about the accuracy differences, I still need a way to check for invalid input data. The upshot is that I had to roll my own isnan and isinf to be able to use -Ofast (because it's actually the underlying __builtin_xxx intrinsics that are broken), which still seems wrong to me.
- xigoi 3y ago
  They are talking about -ffast-math, not -Ofast.
  - gumby 3y ago
    From the gcc manual: -Ofast Disregard strict standards compliance. -Ofast enables all -O3 optimizations. It also enables optimizations that are not valid for all standard-compliant programs. It turns on -ffast-math, -fallow-store-data-races and the Fortran-specific -fstack-arrays, unless -fmax-stack-var-size is specified, and -fno-protect-parens. It turns off -fsemantic-interposition.
jcranmer 3y ago
Here are some of the problems with fast-math: * It links in an object file that enables denormal flushing globally, so that it affects all libraries linked into your application, even if said library explicitly doesn't want fast-math. This is seriously one of the most user-hostile things a compiler can do. * The results of your program will vary depending on the exact make of your compiler and other random attributes of your compile environment, which can wreak havoc if you have code that absolutely wants bit-identical results. This doesn't matter for everybody, but there are some domains where this can be a non-starter (e.g., multiplayer game code). * Fast-math precludes you from being able to use NaN or infinities, and often even being able to defensively test for NaN or infinity. Sure, there are times where this is useful, but an option you might generally prefer to suggest for an uninformed programmer would rather be a "floating-point code can't overflow" option rather than "infinity doesn't exist and it's UB if it does exist". * Fast-math can cause hard range guarantees to fail. Maybe you've got code that you can prove that, even with rounding error, the result will still be >= 0. With fast-math, the code might be adjusted so that the result is instead, say, -1e-10. And if you pass that to a function with a hard domain error at 0 (like sqrt), you now go from the result being 0 to the result being NaN. And see above about what happens when you get NaN. Fast-math is a tradeoff, and if you're willing to except the tradeoff it offers, it's a fine option to use. But most programmers don't even know what the tradeoffs are, and the failure mode can be absolutely catastrophic. It's definitely an option that is in the "you must be this knowledgeable to use" camp.
- dahart 3y ago
  Thank you, great points. I’d have to agree that disabling denorms globally is pretty bad, even if (or maybe especially if) caring about denorms is rare. > Fast-math can cause hard range guarantees to fail. Maybe you’ve got code that you can prove that, even with rounding error, the result will still be >= 0. Floats do this too, it’s pretty routine to bump into epsilon out-of-range issues without fast-math. Most people don’t prove things about their rounding error, and if they do, it’s easy for them to account for 3 ULPs of fast-math error compared to 1/2 ULP for the more accurate operations. Like, nobody who knows what they’re doing will call sqrt() on a number that is fresh out of a multiplier and might be anywhere near zero without testing for zero explicitly, right? I’m sure someone has done it, but I’ve never seen it, and it ranks high on the list of bad ideas even if you steer completely clear of fast-math, no? I guess I just wanted to resist the unspecific parts of the FUD just a little bit. I like your list a lot because it’s specific. Fast-math does carry some additional risks for accuracy sensitive code, and clearly as you and others showed, can infect and impact your whole app, and it can sometimes lead to situations where things break that wouldn’t have happened otherwise. But I think in the grand scheme these situations are quite rare compared to how often people mess up regular floating point math. For a very wide swath of people doing casual arithmetic, fast-math is not likely to cause more problems than floats cause, but it’s fair to want to be careful and pay attention.
light_hue_1 3y ago
Nah, you don't deal with floats. You do machine learning which just happens to use floats. I do both numerical computing and machine learning. And oh boy are you wrong! People who deal with actual numerical computing know that the statement "fast math is only slightly less accurate" is absurd. Fast math is unbounded in its inaccuracy! It can reorder your computations so that something that used to sum to 1 now sums to 0, it can cause catastrophic cancellation, etc. Please stop giving people terrible advice on a topic you're totally unfamiliar with.
- alexey-salmin 3y ago
  > It can reorder your computations so that something that used to sum to 1 now sums to 0, it can cause catastrophic cancellation, etc. Yes, and it could very well be that the correct answer is actually 0 and not 1. Unless you write your code to explicitly account for fp associativity effects, in which case you don't need generic forum advice about fast-math.
- thechao 3y ago
  +1. I'm years away from fp-analysis, but do the transcendental expansions even converge in the presence of fast-math? No `sin()`, no `cos()`, no `exp()`, ...
  - dahart 3y ago
    Well there are library implementations of fast-math trancendentals that offer bounded error, and a million different fast sine approximation algorithms, so, yes? This is why you shouldn’t listen to FUD. The corner cases are indeed frustrating for a few people, but most never hit them, and the world doesn’t suddenly break when fast math is enabled. I am paid to do some FP analysis, btw.
- dahart 3y ago
  I only do numeric computation, I don’t work in machine learning. Sorry your assumptions are incorrect, maybe it’s best not to assume or attack. I didn’t exactly advise using fast math either, I asked for reasoning and pointed out that most casual uses of float aren’t highly sensitive to precision. It’s easy to have wrong sums and catastrophic cancellation without fast math, and it’s relatively rare for fast math to cause those issues when an underlying issue didn’t already exist. I’ve been working in some code that does a couple of quadratic solves and has high order intermediate terms, and I’ve tried using Kahan’s algorithm repeatedly to improve the precision of the discriminants, but it has never helped at all. On the other hand I’ve used a few other tricks that improve the precision enough that the fast math version is higher precision than the naive one without fast math. I get to have my cake and eat it too. Fast math is a tradeoff. Of course it’s a good idea to know what it does and what the risks of using it are, but at least in terms of the accuracy of fast math in CUDA, it’s not an opinion whether the accuracy is relatively close to slow math, it’s reasonably well documented. You can see for yourself that most fast math ops are in the single digit ulps of rounding error. https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#intrinsic-functions https://docs.nvidia.com/cuda/cuda-c-programming-guide/index....

by dahart 3y ago

mort96 3y ago

The scary thing IMO is: your code might be fine with unsafe math optimisations, but maybe you're using a library which is written to do operations in a certain order to minimise numerical error, and unsafe math operations changes the code which are mathematically equivalent but which results in many orders of magnitude more numerical error. It's probably fine most of the time, but it's kinda scary.

deleted 3y ago
[deleted]
dahart 3y ago
It shouldn’t be scary. Any library that is sensitive to order of operations will hopefully have a big fat warning on it. And it can be compiled separately with fast-math disabled. I don’t know of any such libraries off the top of my head, and it’s quite rare to find situations that result in orders of magnitude more error, though I grant you it can happen, and it can be contrived pretty easily.
- planede 3y ago
  You can't fully disable fast-math per-library, moreover a library compiled with fast-math might also introduce inaccuracies in a seemingly unrelated library or application code in the same executable. The reason is that fast-math enables some dynamic initialization of the library that changes the floating point environment in some ways.
  - londons_explore 3y ago
    you're gonna hive to give us a concrete real world example to convince most of us...
  - dahart 3y ago
    > You can’t fully disable fast-math per library Can you elaborate? What fast-math can sneak into a library that disabled fast-math at compile time? > fast-math enables some dynamic initialization of the library that changes the floating point environment in some ways. I wasn’t aware of this, I would love to see some documentation discussing exactly what happens, can you send a link?
    - mort96 3y ago
      > Can you elaborate? What fast-math can sneak into a library that disabled fast-math at compile time? A lot of library code is in headers (especially in C++!). The code in headers is compiled by your compiler using your compile options.
      - dahart 3y ago
        Ah, of course, very good point. A header-only library doesn’t have separate compile options. This is a great reason for a float-sensitive library to not be header-only, right?
        mort96 3y ago
        It's not just about being header-only, lots of libraries which aren't header-only still have code in headers. The library may choose to put certain functions in headers for performance reasons (to let compiler inline them), or, in C++, function templates and class templates generally have to be in headers. But yeah, it's probably a good idea to not put code which breaks under -ffast-math in headers if possible.

usefulcat 3y ago

> Fast math is only slightly less accurate 'slightly'? Last I checked, -Ofast completely breaks std::isnan and std::isinf--they always return false.

dahart 3y ago
Hopefully it was clear from the rest of my comment that I was talking about in-range floats there. I wouldn’t necessarily call inf & nan handling an accuracy issue, that’s more about exceptional cases, but to your point I would have to agree that losing std::isinf is kinda bad since divide by zero is probably near the very top of the list of things most people using floats casually might have to deal with. Which compiler are you using where std::isinf breaks? Hopefully it was also clear that my experience leans toward CUDA, and I think the inf & nan support works there in the presence of NVCC’s fast-math.
- usefulcat 3y ago
  My experience is with gcc and clang on x86. I generally agree with you regarding accuracy, which is why I was quite surprised when I first discovered that -Ofast breaks isnan/isinf. Even if I don't care about the accuracy differences, I still need a way to check for invalid input data. The upshot is that I had to roll my own isnan and isinf to be able to use -Ofast (because it's actually the underlying __builtin_xxx intrinsics that are broken), which still seems wrong to me.
xigoi 3y ago
They are talking about -ffast-math, not -Ofast.
- gumby 3y ago
  From the gcc manual: -Ofast Disregard strict standards compliance. -Ofast enables all -O3 optimizations. It also enables optimizations that are not valid for all standard-compliant programs. It turns on -ffast-math, -fallow-store-data-races and the Fortran-specific -fstack-arrays, unless -fmax-stack-var-size is specified, and -fno-protect-parens. It turns off -fsemantic-interposition.

jcranmer 3y ago

Here are some of the problems with fast-math: * It links in an object file that enables denormal flushing globally, so that it affects all libraries linked into your application, even if said library explicitly doesn't want fast-math. This is seriously one of the most user-hostile things a compiler can do. * The results of your program will vary depending on the exact make of your compiler and other random attributes of your compile environment, which can wreak havoc if you have code that absolutely wants bit-identical results. This doesn't matter for everybody, but there are some domains where this can be a non-starter (e.g., multiplayer game code). * Fast-math precludes you from being able to use NaN or infinities, and often even being able to defensively test for NaN or infinity. Sure, there are times where this is useful, but an option you might generally prefer to suggest for an uninformed programmer would rather be a "floating-point code can't overflow" option rather than "infinity doesn't exist and it's UB if it does exist". * Fast-math can cause hard range guarantees to fail. Maybe you've got code that you can prove that, even with rounding error, the result will still be >= 0. With fast-math, the code might be adjusted so that the result is instead, say, -1e-10. And if you pass that to a function with a hard domain error at 0 (like sqrt), you now go from the result being 0 to the result being NaN. And see above about what happens when you get NaN. Fast-math is a tradeoff, and if you're willing to except the tradeoff it offers, it's a fine option to use. But most programmers don't even know what the tradeoffs are, and the failure mode can be absolutely catastrophic. It's definitely an option that is in the "you must be this knowledgeable to use" camp.

dahart 3y ago
Thank you, great points. I’d have to agree that disabling denorms globally is pretty bad, even if (or maybe especially if) caring about denorms is rare. > Fast-math can cause hard range guarantees to fail. Maybe you’ve got code that you can prove that, even with rounding error, the result will still be >= 0. Floats do this too, it’s pretty routine to bump into epsilon out-of-range issues without fast-math. Most people don’t prove things about their rounding error, and if they do, it’s easy for them to account for 3 ULPs of fast-math error compared to 1/2 ULP for the more accurate operations. Like, nobody who knows what they’re doing will call sqrt() on a number that is fresh out of a multiplier and might be anywhere near zero without testing for zero explicitly, right? I’m sure someone has done it, but I’ve never seen it, and it ranks high on the list of bad ideas even if you steer completely clear of fast-math, no? I guess I just wanted to resist the unspecific parts of the FUD just a little bit. I like your list a lot because it’s specific. Fast-math does carry some additional risks for accuracy sensitive code, and clearly as you and others showed, can infect and impact your whole app, and it can sometimes lead to situations where things break that wouldn’t have happened otherwise. But I think in the grand scheme these situations are quite rare compared to how often people mess up regular floating point math. For a very wide swath of people doing casual arithmetic, fast-math is not likely to cause more problems than floats cause, but it’s fair to want to be careful and pay attention.

light_hue_1 3y ago

Nah, you don't deal with floats. You do machine learning which just happens to use floats. I do both numerical computing and machine learning. And oh boy are you wrong! People who deal with actual numerical computing know that the statement "fast math is only slightly less accurate" is absurd. Fast math is unbounded in its inaccuracy! It can reorder your computations so that something that used to sum to 1 now sums to 0, it can cause catastrophic cancellation, etc. Please stop giving people terrible advice on a topic you're totally unfamiliar with.

alexey-salmin 3y ago
> It can reorder your computations so that something that used to sum to 1 now sums to 0, it can cause catastrophic cancellation, etc. Yes, and it could very well be that the correct answer is actually 0 and not 1. Unless you write your code to explicitly account for fp associativity effects, in which case you don't need generic forum advice about fast-math.
thechao 3y ago
+1. I'm years away from fp-analysis, but do the transcendental expansions even converge in the presence of fast-math? No `sin()`, no `cos()`, no `exp()`, ...
- dahart 3y ago
  Well there are library implementations of fast-math trancendentals that offer bounded error, and a million different fast sine approximation algorithms, so, yes? This is why you shouldn’t listen to FUD. The corner cases are indeed frustrating for a few people, but most never hit them, and the world doesn’t suddenly break when fast math is enabled. I am paid to do some FP analysis, btw.
dahart 3y ago
I only do numeric computation, I don’t work in machine learning. Sorry your assumptions are incorrect, maybe it’s best not to assume or attack. I didn’t exactly advise using fast math either, I asked for reasoning and pointed out that most casual uses of float aren’t highly sensitive to precision. It’s easy to have wrong sums and catastrophic cancellation without fast math, and it’s relatively rare for fast math to cause those issues when an underlying issue didn’t already exist. I’ve been working in some code that does a couple of quadratic solves and has high order intermediate terms, and I’ve tried using Kahan’s algorithm repeatedly to improve the precision of the discriminants, but it has never helped at all. On the other hand I’ve used a few other tricks that improve the precision enough that the fast math version is higher precision than the naive one without fast math. I get to have my cake and eat it too. Fast math is a tradeoff. Of course it’s a good idea to know what it does and what the risks of using it are, but at least in terms of the accuracy of fast math in CUDA, it’s not an opinion whether the accuracy is relatively close to slow math, it’s reasonably well documented. You can see for yourself that most fast math ops are in the single digit ulps of rounding error. https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#intrinsic-functions https://docs.nvidia.com/cuda/cuda-c-programming-guide/index....