4 ms·

I've read that, if one is going to use floats, then one should eschew 32-bit floats, and go with 64-bit floats. 80-bit floats seem only useful for very special

I've read that, if one is going to use floats, then one should eschew 32-bit floats, and go with 64-bit floats. 80-bit floats seem only useful for very special applications. I converted a satellite orbital prediction program that used doubles (64-bit floats) to use long doubles (80-bit floats) but only saw a difference in the 11th decimal place, with lots of sin/cos/atan/sqrt involved in the calculations. With 32-bit floats it was a joke: the accumulated error made the 'predictions' useless. The basic problem with 32-bit is you have to be quite careful with your arithmetic so you don't lose significance. It's almost as bad as using Q-notation. https://en.wikipedia.org/wiki/Q_(number_format) https://en.wikipedia.org/wiki/Q_(number_format)

dynm 5y ago
The main use for 80-bit floats is probably to confirm that when you switch from 64-bit to 80-bit, nothing changes. This is a very important use, though!
dragontamer 5y ago
Errors grow exponentially in most floating-point operations. The exception is cancellation error, which can suddenly cause the error rate to spike dramatically. But if you sort your numbers before adding/subtracting, you can often negate this overwhelming cancellation error. The exponential growth of other floating point errors is almost unstoppable however. The only real way to stop it is to design an algorithm that operates with fewer steps. --------- 64-bit has a much smaller initial error (1/2^53), while 32-bit has (1/2^23) initial errors... starting with 30-magnitudes more error than double-precision.
- vlovich123 5y ago
  Kahan summation[1] may be better than just doing a sorted sum & doesn't require you have all the numbers up-front to sort (i.e. running sum). Although I think even it benefits from doing it in sorted order if you do have the full history available & the wikipedia article lists a parallelization technique you can leverage in that case. Klein sum might be better but I haven't investigated. EDIT: Apparently there's a nice Rust crate for doing this stuff: https://crates.io/crates/accurate https://crates.io/crates/accurate [1] https://en.wikipedia.org/wiki/Kahan_summation_algorithm#Alternatives https://en.wikipedia.org/wiki/Kahan_summation_algorithm#Alte...
  - adgjlsfhk1 5y ago
    One thing to consider here is XSum. It's a C library that has surprisingly fast exact floating point sums.
  - a1369209993 5y ago
    Kahan summation is basically just synthesizing a larger floating point type (with 46/106 bits of mantissa for 32/64-bit base type) and then using that to do the sum, though. You still lose precision if your errors exceed that new, larger margin. Eg try adding 34000 + 56 + .078 - 56 - 34000, carrying two digits per float.
modeless 5y ago
32-bit floats are sufficient and desired for performance in many cases, like graphics or audio. 64-bit floats are not enough in some cases, like galaxy-scale coordinates or nanosecond precision over long time periods or some currency calculations. There's no single best answer, you have to know what your requirements are.
- retsibsi 5y ago
  > 32-bit floats are sufficient and desired for performance in many cases, like graphics or audio I was googling this recently, when deciding whether to use floats or doubles in a simple game I was writing. Some answers suggested that (32-bit) floats aren't necessarily any faster in real-world applications on modern hardware. If you or anyone else feels like elaborating on this I'd be interested.
  - gpderetta 5y ago
    For scalar computations 32 bit floats are not going to be faster except maybe divisions. There are second order effects due the reduced memory usage. For anything that is vectorizable or otherwise amenable to SIMD optimizations you can get up to 2x speedup.
    - adgjlsfhk1 5y ago
      All the transcendental functions are also an extra 2-4x slower for Float64 since you need higher order approximations, and the compensated arithmetic is slower.
  - zamadatix 5y ago
    For CPU cycle counts it's not that different, for GPU cycle counts it's something in the range of 50x FLOPS difference for consumer GPUs so avoid it in shaders and whatnot if you can. There is also the matter of SIMD on the CPU, you can generally get twice the SIMD rate by halving the bit width of the data which is quite the gain. Outside raw computation rate though there is also the matter of cache and memory bandwidth. You'll use 2x with doubles which can really bog things down as even if the CPU can crunch the calculation quickly it could spend a lot of time waiting for the data to crunch to arrive. All that being said if you know your game is going to be simple then it probably doesn't matter one way or the other even if it does come at a big performance cost.
zokier 5y ago
> I've read that, if one is going to use floats, then one should eschew 32-bit floats, and go with 64-bit floats. 80-bit floats seem only useful for very special applications. This is what Kahan has to say on the matter: For now the 10-byte Extended format is a tolerable compromise between the value of extra-precise arithmetic and the price of implementing it to run fast; very soon two more bytes of precision will become tolerable, and ultimately a 16-byte format ... That kind of gradual evolution towards wider precision was already in view when IEEE Standard 754 for Floating-Point Arithmetic was framed (The quote is apparently from 1994 unpublished manuscript, so I can't find the source for it) More than 25 years later and we are rarely using even 80 bit doubles, nevermind 128 bit ones.
- Narishma 5y ago
  Isn't that because 80-bit floats have been deprecated a decade or two ago in x86, and were never implemented in other architectures to begin with?
  - perl4ever 5y ago
    Both architectures implemented 80-bit floats - Intel x86 and Motorola 68k.
  - adgjlsfhk1 5y ago
    They were deprecated in x86 because they weren't widely used.
dev_tty01 5y ago
Depends on what you are doing. 8-bit floats are widely used in large neural network applications. (No that is not a typo. I mean 8-bit, not 80-bit.)
- adgjlsfhk1 5y ago
  Really? I thought that for 8 bits, fixed point formats were much more common.
- ant6n 5y ago
  At that point, it would seem more sensible to just define a list of values that are represented by each possible 8-bit number, and use LUTs for operations. Why even torture this exponent and mantissa concept down to 8 bits when one can have a domain specific set of representable numbers that have exactly the properties that are needed.
  - GeorgeTirebiter 5y ago
    I did exactly this - 8-bit floats. The application needed fairly accurate representations around 0; but didn't wildly swing. It was for an IMU that had to have the internal representation converted to 'regular units'. Ultimately, Q.15 won; but it was interesting. In fact, I encourage you to try it with e.g. 4 mantissa 4 exponent, and enumerate all values.
    - ant6n 5y ago
      For a game boy (original) project I wanted to represent distance values in range 0.3..32 with a u8, but with higher precision for closer values, but approximately linear relationship for distances that are close to one another, with precision of ~0.05 around d=0 and <0.30 around d=32. I ended up creating a set of values that follows different mathematical functions in different segments (exponential, linear, quadratic). As a float this wouldn´t have worked, because the precision halves for every new exponent, so the distances between adjacent values are not approximately linear.
alfalfasprout 5y ago
It really depends what you're using the floats for. For ML applications? 16 bit floats do the job with some trickery (quantization). For scientific computing? Generally 64 bit floats give you the needed precision. But if you're chaining a lot of operations that exacerbate floating point error you may need more. For usage in trading? Again, depends on what computations you're doing. There's no "hard and fast" rule.

dynm 5y ago

The main use for 80-bit floats is probably to confirm that when you switch from 64-bit to 80-bit, nothing changes. This is a very important use, though!

dragontamer 5y ago

Errors grow exponentially in most floating-point operations. The exception is cancellation error, which can suddenly cause the error rate to spike dramatically. But if you sort your numbers before adding/subtracting, you can often negate this overwhelming cancellation error. The exponential growth of other floating point errors is almost unstoppable however. The only real way to stop it is to design an algorithm that operates with fewer steps. --------- 64-bit has a much smaller initial error (1/2^53), while 32-bit has (1/2^23) initial errors... starting with 30-magnitudes more error than double-precision.

vlovich123 5y ago
Kahan summation[1] may be better than just doing a sorted sum & doesn't require you have all the numbers up-front to sort (i.e. running sum). Although I think even it benefits from doing it in sorted order if you do have the full history available & the wikipedia article lists a parallelization technique you can leverage in that case. Klein sum might be better but I haven't investigated. EDIT: Apparently there's a nice Rust crate for doing this stuff: https://crates.io/crates/accurate https://crates.io/crates/accurate [1] https://en.wikipedia.org/wiki/Kahan_summation_algorithm#Alternatives https://en.wikipedia.org/wiki/Kahan_summation_algorithm#Alte...
- adgjlsfhk1 5y ago
  One thing to consider here is XSum. It's a C library that has surprisingly fast exact floating point sums.
- a1369209993 5y ago
  Kahan summation is basically just synthesizing a larger floating point type (with 46/106 bits of mantissa for 32/64-bit base type) and then using that to do the sum, though. You still lose precision if your errors exceed that new, larger margin. Eg try adding 34000 + 56 + .078 - 56 - 34000, carrying two digits per float.

modeless 5y ago

32-bit floats are sufficient and desired for performance in many cases, like graphics or audio. 64-bit floats are not enough in some cases, like galaxy-scale coordinates or nanosecond precision over long time periods or some currency calculations. There's no single best answer, you have to know what your requirements are.

retsibsi 5y ago
> 32-bit floats are sufficient and desired for performance in many cases, like graphics or audio I was googling this recently, when deciding whether to use floats or doubles in a simple game I was writing. Some answers suggested that (32-bit) floats aren't necessarily any faster in real-world applications on modern hardware. If you or anyone else feels like elaborating on this I'd be interested.
- gpderetta 5y ago
  For scalar computations 32 bit floats are not going to be faster except maybe divisions. There are second order effects due the reduced memory usage. For anything that is vectorizable or otherwise amenable to SIMD optimizations you can get up to 2x speedup.
  - adgjlsfhk1 5y ago
    All the transcendental functions are also an extra 2-4x slower for Float64 since you need higher order approximations, and the compensated arithmetic is slower.
- zamadatix 5y ago
  For CPU cycle counts it's not that different, for GPU cycle counts it's something in the range of 50x FLOPS difference for consumer GPUs so avoid it in shaders and whatnot if you can. There is also the matter of SIMD on the CPU, you can generally get twice the SIMD rate by halving the bit width of the data which is quite the gain. Outside raw computation rate though there is also the matter of cache and memory bandwidth. You'll use 2x with doubles which can really bog things down as even if the CPU can crunch the calculation quickly it could spend a lot of time waiting for the data to crunch to arrive. All that being said if you know your game is going to be simple then it probably doesn't matter one way or the other even if it does come at a big performance cost.

zokier 5y ago

> I've read that, if one is going to use floats, then one should eschew 32-bit floats, and go with 64-bit floats. 80-bit floats seem only useful for very special applications. This is what Kahan has to say on the matter: For now the 10-byte Extended format is a tolerable compromise between the value of extra-precise arithmetic and the price of implementing it to run fast; very soon two more bytes of precision will become tolerable, and ultimately a 16-byte format ... That kind of gradual evolution towards wider precision was already in view when IEEE Standard 754 for Floating-Point Arithmetic was framed (The quote is apparently from 1994 unpublished manuscript, so I can't find the source for it) More than 25 years later and we are rarely using even 80 bit doubles, nevermind 128 bit ones.

Narishma 5y ago
Isn't that because 80-bit floats have been deprecated a decade or two ago in x86, and were never implemented in other architectures to begin with?
- perl4ever 5y ago
  Both architectures implemented 80-bit floats - Intel x86 and Motorola 68k.
- adgjlsfhk1 5y ago
  They were deprecated in x86 because they weren't widely used.

dev_tty01 5y ago

Depends on what you are doing. 8-bit floats are widely used in large neural network applications. (No that is not a typo. I mean 8-bit, not 80-bit.)

adgjlsfhk1 5y ago
Really? I thought that for 8 bits, fixed point formats were much more common.
ant6n 5y ago
At that point, it would seem more sensible to just define a list of values that are represented by each possible 8-bit number, and use LUTs for operations. Why even torture this exponent and mantissa concept down to 8 bits when one can have a domain specific set of representable numbers that have exactly the properties that are needed.
- GeorgeTirebiter 5y ago
  I did exactly this - 8-bit floats. The application needed fairly accurate representations around 0; but didn't wildly swing. It was for an IMU that had to have the internal representation converted to 'regular units'. Ultimately, Q.15 won; but it was interesting. In fact, I encourage you to try it with e.g. 4 mantissa 4 exponent, and enumerate all values.
  - ant6n 5y ago
    For a game boy (original) project I wanted to represent distance values in range 0.3..32 with a u8, but with higher precision for closer values, but approximately linear relationship for distances that are close to one another, with precision of ~0.05 around d=0 and <0.30 around d=32. I ended up creating a set of values that follows different mathematical functions in different segments (exponential, linear, quadratic). As a float this wouldn´t have worked, because the precision halves for every new exponent, so the distances between adjacent values are not approximately linear.

alfalfasprout 5y ago

It really depends what you're using the floats for. For ML applications? 16 bit floats do the job with some trickery (quantization). For scientific computing? Generally 64 bit floats give you the needed precision. But if you're chaining a lot of operations that exacerbate floating point error you may need more. For usage in trading? Again, depends on what computations you're doing. There's no "hard and fast" rule.