4 ms·

So half the complaint here is that Java doesn't let you use the 80-bit x87 floating-point type. Well, actually it does--if you don't add `strictfp` to your meth

So half the complaint here is that Java doesn't let you use the 80-bit x87 floating-point type. Well, actually it does--if you don't add `strictfp` to your method declarations, Java is allowed to compute intermediate results of float and double computations in higher precision, and this was retrofitted because it's actually somewhat difficult to get the x87 floating-point unit to actually do single- or double-precision (as opposed to extended-precision) arithmetic. But you don't actually want to use the 80-bit type; it's slower in practice, and it has some extra wonkiness over other IEEE 754 types that it's really not worth it. And on 64-bit x86, everyone just uses SSE for floating-point units and thus you don't need to worry about x87 unless you explicitly opt into its insanity. As a numerical analyst, Kahan is pretty obsessed with use-as-much-precision-as-you-can. But there's a useful rule of thumb: you need about twice the amount of working precision as your final result. Since double precision has a 53-bit mantissa (~16 decimal digits), that means if you need only 8 or fewer decimal digits, you're completely fine with double precision. And furthermore, the experiences I've had with many programmers suggest that getting bit-equivalent results from different machines is a higher priority than squeezing the best possible numerics out of your hardware. HPC does tend to care about the latter a lot more, but that's also an area where the solution is almost always to just use your system's advanced math libraries (e.g., MKL for dense linear algebra).

thebooktocome 3y ago
Strictfp is no longer used as of JVM 17. IEEE-754 semantics are default. https://openjdk.org/jeps/306 https://openjdk.org/jeps/306 Ironically, using your system’s math libraries will probably make replicating floating point results harder. Especially on macOS. Accelerate is a curse. I disagree with your gloss of Kahan’s philosophy. His approach is more along the lines of “do not waste precision”. But this philosophy is not the complete truth; as close as I can state it briefly, my modification would be “do not waste precision that may be needed later”.
- jcranmer 3y ago
  I was unaware that Java finally got rid of strictfp (though happy that it finally did). It was added in Java 1.2, perhaps in response to this paper, though I don't know the entire timeline accurately.
- arthur2e5 3y ago
  > It is not a goal to define any sort of "fast-fp" or "loose-fp" (c.f. JSR 84: Floating Point Extensions). (Comic villain sitting in his fast-math lair) Foiled yet again!
  - thebooktocome 3y ago
    As a fellow practitioner of the demonic -ffast-math arts, let us cackle menacingly together. :)
    - a_e_k 3y ago
      I find that -ffast-math is not so bad, so long as I develop and test with it from the beginning. It's much like any of the other more aggressive optimizations in that sense. Plus, comparing against strict math as I go tends to highlight where I might have been about to do something dodgy anyway.
    - lifthrasiir 3y ago
      Did you mean -fbroken-and-not-necessarily-fast-math? [1] [1] But really, if -ffast-math does turn -funsafe-math-optimizations on, it should have been named similarly. There is a possibility of much safer -ffast-math with almost zero breakage (by assuming a subset of IEEE 754, like the fixed rounding mode). The current -ffast-math is so reckless [2]. [2] https://simonbyrne.github.io/notes/fastmath/#flushing_subnormals_to_zero https://simonbyrne.github.io/notes/fastmath/#flushing_subnor...
  - jcranmer 3y ago
    I've long since decided that my write up on floating point will be titled "Floating Point or: How I Learned to Start Worrying and Hate Fast-Math".
- kragen 3y ago
  though you imply the contrary, the 8087 is ieee-754-compliant; kahan's involvement in its design and in the standards process ensured that
  - thebooktocome 3y ago
    Where did I imply that? The fault was with the JVM. “The impetus for changing the default floating-point semantics of the platform in the late 1990's stemmed from a bad interaction between the original Java language and JVM semantics and some unfortunate peculiarities of the x87 floating-point co-processor instruction set of the popular x86 architecture.”
    - kragen 3y ago
      glad we've cleared that up
    - titzer 3y ago
      The x87 ISA was an absolutely boondoggle which took 3 decades to recover from. It was so clearly not designed by CPU architects. What a disaster.
- owlbite 3y ago
  Attempting to get bit-exact reproducible results across different hardware is a fool's errand (if you care in the least about performance). The nature of the beast is that as soon as you change the order of arithmetic you're going to get a different result. Optimized code is going to give you different results on different hardware due to the fact that you need to optimize things differently. Threading, memory alignment and/or different versions of the library software are likely to lead to different results even on the same machine unless the authors of the library go out of the way to promise repeatability. (If you want to get the same answer, run on a single thread, page align everything you feed in, and never upgrade your system; alternatively write a scalar loop in C, compile with -O0 and pray the compiler doesn't change the order of things on its next upgrade).
  - MauranKilom 3y ago
    ...or use fixed-point arithmetic. Which, if I understand correctly, is basically the go-to of modern multiplayer-enabled game engines.
    - pajko 3y ago
      Or posits: https://en.wikipedia.org/wiki/Unum_(number_format) https://en.wikipedia.org/wiki/Unum_(number_format)
  - titzer 3y ago
    > Attempting to get bit-exact reproducible results across different hardware is a fool's errand (if you care in the least about performance). We did it for Wasm, which follows IEEE-754 semantics exactly for 32-bit and 64-bit floats. (The only nondeterminism is the exact bit pattern you get for NaNs in some circumstances.) Rounding is 100% well-specified. And CPUs have done that for decades. Even vector ISAs have learned that non-IEEE results are not what software wants; all vector ISAs are converging on IEEE-754. > Optimized code is going to give you different results on different hardware due to the fact that you need to optimize things differently. This is due to C/C++ (and to some extent Fortran) semantics. It is not hardware. What do threads have to do with floating point precision?
mkl 3y ago
> But there's a useful rule of thumb: you need about twice the amount of working precision as your final result. Since double precision has a 53-bit mantissa (~16 decimal digits), that means if you need only 8 or fewer decimal digits, you're completely fine with double precision. This doesn't seem right, or at least it's not very general. The more operations you do, the more rounding errors you have, and each operation has the potential to magnify earlier errors. In solving ill-conditioned problems (which are not uncommon) the errors can easily be magnified so much that they're bigger than your signal, even with relatively small and simple situations.
ClumsyPilot 3y ago
> precision has a 53-bit mantissa (~16 decimal digits), that means if you need only 8 or fewer decimal digits, you're completely fine with double precision phave you ever played Kerbal Soace Program? have you met the Kraken? the game simulates a spacecraft you build piece by piece. So you are flying around in space and a thruster fires and maybe some part of the ship is poorly attached and it wobbling. Everything is fine, but then you clip upper atmosphere of mars, and everything goes to shit - tleven though it shoupd pose no threat, the spacecraft spontaneously shakes itself to pieces That game is plagued by massove problems with floating point errors , they kill you vrew, they ruin your missions, your speed of rotation becomea a NAN
- imtringued 3y ago
  OpenSpace uses 32 bit floating point numbers and can scale from planck length to the whole diameter of the observable universe with practically no loss of precision. The problem with this approach is that KSP would need to rewrite the entire rendering engine and physics engine from scratch to support infinite precision.
- crote 3y ago
  KSP's Kraken pretty much proves their point, no? Simulating human-scale physics in a solar-system-scale world needs far more than 8 decimal digits. Neptune's orbit is 12 orders of magnitude larger than your spacecraft, no wonder your floats are acting up. You're going to need at least 50-bit precision to make your simulations accurate to the millimeter. If anything, the fact that KSP is pretty much the only game with serious float issues shows that doubles are just fine for most applications.

thebooktocome 3y ago

Strictfp is no longer used as of JVM 17. IEEE-754 semantics are default. https://openjdk.org/jeps/306 https://openjdk.org/jeps/306 Ironically, using your system’s math libraries will probably make replicating floating point results harder. Especially on macOS. Accelerate is a curse. I disagree with your gloss of Kahan’s philosophy. His approach is more along the lines of “do not waste precision”. But this philosophy is not the complete truth; as close as I can state it briefly, my modification would be “do not waste precision that may be needed later”.

jcranmer 3y ago
I was unaware that Java finally got rid of strictfp (though happy that it finally did). It was added in Java 1.2, perhaps in response to this paper, though I don't know the entire timeline accurately.
arthur2e5 3y ago
> It is not a goal to define any sort of "fast-fp" or "loose-fp" (c.f. JSR 84: Floating Point Extensions). (Comic villain sitting in his fast-math lair) Foiled yet again!
- thebooktocome 3y ago
  As a fellow practitioner of the demonic -ffast-math arts, let us cackle menacingly together. :)
  - a_e_k 3y ago
    I find that -ffast-math is not so bad, so long as I develop and test with it from the beginning. It's much like any of the other more aggressive optimizations in that sense. Plus, comparing against strict math as I go tends to highlight where I might have been about to do something dodgy anyway.
  - lifthrasiir 3y ago
    Did you mean -fbroken-and-not-necessarily-fast-math? [1] [1] But really, if -ffast-math does turn -funsafe-math-optimizations on, it should have been named similarly. There is a possibility of much safer -ffast-math with almost zero breakage (by assuming a subset of IEEE 754, like the fixed rounding mode). The current -ffast-math is so reckless [2]. [2] https://simonbyrne.github.io/notes/fastmath/#flushing_subnormals_to_zero https://simonbyrne.github.io/notes/fastmath/#flushing_subnor...
- jcranmer 3y ago
  I've long since decided that my write up on floating point will be titled "Floating Point or: How I Learned to Start Worrying and Hate Fast-Math".
kragen 3y ago
though you imply the contrary, the 8087 is ieee-754-compliant; kahan's involvement in its design and in the standards process ensured that
- thebooktocome 3y ago
  Where did I imply that? The fault was with the JVM. “The impetus for changing the default floating-point semantics of the platform in the late 1990's stemmed from a bad interaction between the original Java language and JVM semantics and some unfortunate peculiarities of the x87 floating-point co-processor instruction set of the popular x86 architecture.”
  - kragen 3y ago
    glad we've cleared that up
  - titzer 3y ago
    The x87 ISA was an absolutely boondoggle which took 3 decades to recover from. It was so clearly not designed by CPU architects. What a disaster.
owlbite 3y ago
Attempting to get bit-exact reproducible results across different hardware is a fool's errand (if you care in the least about performance). The nature of the beast is that as soon as you change the order of arithmetic you're going to get a different result. Optimized code is going to give you different results on different hardware due to the fact that you need to optimize things differently. Threading, memory alignment and/or different versions of the library software are likely to lead to different results even on the same machine unless the authors of the library go out of the way to promise repeatability. (If you want to get the same answer, run on a single thread, page align everything you feed in, and never upgrade your system; alternatively write a scalar loop in C, compile with -O0 and pray the compiler doesn't change the order of things on its next upgrade).
- MauranKilom 3y ago
  ...or use fixed-point arithmetic. Which, if I understand correctly, is basically the go-to of modern multiplayer-enabled game engines.
  - pajko 3y ago
    Or posits: https://en.wikipedia.org/wiki/Unum_(number_format) https://en.wikipedia.org/wiki/Unum_(number_format)
- titzer 3y ago
  > Attempting to get bit-exact reproducible results across different hardware is a fool's errand (if you care in the least about performance). We did it for Wasm, which follows IEEE-754 semantics exactly for 32-bit and 64-bit floats. (The only nondeterminism is the exact bit pattern you get for NaNs in some circumstances.) Rounding is 100% well-specified. And CPUs have done that for decades. Even vector ISAs have learned that non-IEEE results are not what software wants; all vector ISAs are converging on IEEE-754. > Optimized code is going to give you different results on different hardware due to the fact that you need to optimize things differently. This is due to C/C++ (and to some extent Fortran) semantics. It is not hardware. What do threads have to do with floating point precision?

mkl 3y ago

> But there's a useful rule of thumb: you need about twice the amount of working precision as your final result. Since double precision has a 53-bit mantissa (~16 decimal digits), that means if you need only 8 or fewer decimal digits, you're completely fine with double precision. This doesn't seem right, or at least it's not very general. The more operations you do, the more rounding errors you have, and each operation has the potential to magnify earlier errors. In solving ill-conditioned problems (which are not uncommon) the errors can easily be magnified so much that they're bigger than your signal, even with relatively small and simple situations.

ClumsyPilot 3y ago

> precision has a 53-bit mantissa (~16 decimal digits), that means if you need only 8 or fewer decimal digits, you're completely fine with double precision phave you ever played Kerbal Soace Program? have you met the Kraken? the game simulates a spacecraft you build piece by piece. So you are flying around in space and a thruster fires and maybe some part of the ship is poorly attached and it wobbling. Everything is fine, but then you clip upper atmosphere of mars, and everything goes to shit - tleven though it shoupd pose no threat, the spacecraft spontaneously shakes itself to pieces That game is plagued by massove problems with floating point errors , they kill you vrew, they ruin your missions, your speed of rotation becomea a NAN

imtringued 3y ago
OpenSpace uses 32 bit floating point numbers and can scale from planck length to the whole diameter of the observable universe with practically no loss of precision. The problem with this approach is that KSP would need to rewrite the entire rendering engine and physics engine from scratch to support infinite precision.
crote 3y ago
KSP's Kraken pretty much proves their point, no? Simulating human-scale physics in a solar-system-scale world needs far more than 8 decimal digits. Neptune's orbit is 12 orders of magnitude larger than your spacecraft, no wonder your floats are acting up. You're going to need at least 50-bit precision to make your simulations accurate to the millimeter. If anything, the fact that KSP is pretty much the only game with serious float issues shows that doubles are just fine for most applications.