5 ms·

I've always wondered how much is wasted supporting the full IEEE floating point standard. How rare is it for a program to actually care about rounding behavior

by tgb 9y ago

I've always wondered how much is wasted supporting the full IEEE floating point standard. How rare is it for a program to actually care about rounding behavior and NaN and inf and whatever that support for tiny numbers is called? Obviously medical imaging devices and physics simulations will need to stop and do this properly but they can get special hardware. How much silicon goes towards supporting that stuff in an i7? In a 1080? I have no idea, but it sounds like a lot to me.

jimrandomh 9y ago
Consistent rounding behavior matters when you need reproducibility. For example, videogames often reproduce computations as part of their networking systems, and if different users have different rounding behavior, it'll break.
- eigenbom 9y ago
  fixed point is often used for that case
kdkkfcg 9y ago
So, if you read the article you will learn the float handling is done in software.
- flavio81 9y ago
  Yes... but note that the final machine language code does use the processor opcodes that handle floats: MULSS, MOVSS http://www.felixcloutier.com/x86/MULSS.html http://www.felixcloutier.com/x86/MULSS.html http://www.felixcloutier.com/x86/MOVSS.html http://www.felixcloutier.com/x86/MOVSS.html So the final float multiplication is done by the hardware. What the software does, is to (a) handle the complex number and convert it to floats, and probably (b) handle the double precision using single precision x86 opcodes.
- tgb 9y ago
  Yes, I read the article and it inspired my question.
taeric 9y ago
I find this harkens well to a lot of heavily modeled frameworks. How many people truly need everything you get with the latest React/etc? I realize these aren't directly comparable. However, it is hilariously depressing how many "dynamic" sites would be better served by a simple static html file.
- fsloth 9y ago
  That's not really the same. IEEE 754 (https://en.m.wikipedia.org/wiki/IEEE_754 https://en.m.wikipedia.org/wiki/IEEE_754) defines the algebra for two types - the single and double precision floats. Claiming you don't need all of it means analogically you would be fine with a car that turns only left and explodes on tuesdays. Which might be true for your case, but an abysmal set of limitations for a general purpose device.
  - taeric 9y ago
    I acknowledged they aren't directly comparable. To the point that I would actually leverage the question the other direction for most things. Making sure arithmetic is incredibly well defined seems a more important point than what most extension points for frameworks focus on. (Which, I think, is what you are saying. So that I think we are in agreement. :) )
planteen 9y ago
Rounding behavior is important for more than medical imaging and physics simulations. Many embedded systems controlling or sensing something in the real world care about it. Not just things like cars and planes, but imagine control loops designed for up-times of years that suddenly serious numerical stability issues because part of the IEEE standard got thrown out.
comicjk 9y ago
I do physics simulations and I generally use -ffast-math. For most simulation types, proper algorithms are stable with respect to small floating-point errors (this has been a subject of research for decades). The extra speed provided by imprecise math allows for bigger, longer simulations, which ultimately means more accuracy than what is lost in the floating-point operations.
- tgb 9y ago
  That's really interesting, thanks!
fsloth 9y ago
Wasted? Well, if you don't care for video editing, image processing, 3D modeling... A well defined floating point implementation is everyones benefit.
stinos 9y ago
I don't have an idea of how much it matters for which percentage of programs, but here are some anecdotes of what can happen when not caring about NaN and rounding. We had a bug once which was really hard to track down. Turned out a simple lack of initialization of a variable in a simple piece of code for a running average: the programmer reasoned in the first iteration this number is going to be multiplied with 0.0 anyway, so no need to initialize it. However because there are many NaN representations, once in a while the uninitialized memory of the variable happened to represent NaN, and NaN * 0.0 is still NaN, and that would keep on propagating, oops. Likewise for rounding: we had some code calculating pixel positions and sometimes stuff was one pixel off resulting in weird visual artefacts. Turns out that when feeding the (faulty) rounding algorithm with results of some floating point calculations wouldn't do the expected thing but round down instead of up or so. Oops.
MrQuincle 9y ago
I was running MCMC for a while just in probability space. That didn't last long! It's tremendously useful to work in log probability space. It adds tremendous expressivity close to zero plus makes most multiplications sums. I think similar methods are probably used in other sectors. What I found remarkable though was the big difference I saw in old versus new compilers with respect to the diversity of random numbers generated. I first thought that it was the entropy available on my system but that didn't turn out to be the key factor. So watch out for old compilers!
s__mac 9y ago
The bummer about this to me is that the c99 guys took it upon themselves to extend IEEE semantics to complex numbers. To do that, you have to decide on a representation for infinity in complex numbers (there's really only one mathematically). They choose isinf(re(x)) || isinf(im(x)), so among other things, Inf + Nan*i is considered infinite...ok. And then they tried to make things consistent between real and complex numbers with this (made up) notion of complex infinity. The result being that there's edge cases they have to back up and 'fix' to be consistent. However, they could have just decided that complex arithmetic semantics follow naturally from floating point and whatever results ensue. So in the end this was all self-inflicted.

by tgb 9y ago

jimrandomh 9y ago

Consistent rounding behavior matters when you need reproducibility. For example, videogames often reproduce computations as part of their networking systems, and if different users have different rounding behavior, it'll break.

eigenbom 9y ago
fixed point is often used for that case

kdkkfcg 9y ago

So, if you read the article you will learn the float handling is done in software.

flavio81 9y ago
Yes... but note that the final machine language code does use the processor opcodes that handle floats: MULSS, MOVSS http://www.felixcloutier.com/x86/MULSS.html http://www.felixcloutier.com/x86/MULSS.html http://www.felixcloutier.com/x86/MOVSS.html http://www.felixcloutier.com/x86/MOVSS.html So the final float multiplication is done by the hardware. What the software does, is to (a) handle the complex number and convert it to floats, and probably (b) handle the double precision using single precision x86 opcodes.
tgb 9y ago
Yes, I read the article and it inspired my question.

taeric 9y ago

I find this harkens well to a lot of heavily modeled frameworks. How many people truly need everything you get with the latest React/etc? I realize these aren't directly comparable. However, it is hilariously depressing how many "dynamic" sites would be better served by a simple static html file.

fsloth 9y ago
That's not really the same. IEEE 754 (https://en.m.wikipedia.org/wiki/IEEE_754 https://en.m.wikipedia.org/wiki/IEEE_754) defines the algebra for two types - the single and double precision floats. Claiming you don't need all of it means analogically you would be fine with a car that turns only left and explodes on tuesdays. Which might be true for your case, but an abysmal set of limitations for a general purpose device.
- taeric 9y ago
  I acknowledged they aren't directly comparable. To the point that I would actually leverage the question the other direction for most things. Making sure arithmetic is incredibly well defined seems a more important point than what most extension points for frameworks focus on. (Which, I think, is what you are saying. So that I think we are in agreement. :) )

planteen 9y ago

Rounding behavior is important for more than medical imaging and physics simulations. Many embedded systems controlling or sensing something in the real world care about it. Not just things like cars and planes, but imagine control loops designed for up-times of years that suddenly serious numerical stability issues because part of the IEEE standard got thrown out.

comicjk 9y ago

I do physics simulations and I generally use -ffast-math. For most simulation types, proper algorithms are stable with respect to small floating-point errors (this has been a subject of research for decades). The extra speed provided by imprecise math allows for bigger, longer simulations, which ultimately means more accuracy than what is lost in the floating-point operations.

tgb 9y ago
That's really interesting, thanks!

fsloth 9y ago

Wasted? Well, if you don't care for video editing, image processing, 3D modeling... A well defined floating point implementation is everyones benefit.

stinos 9y ago

I don't have an idea of how much it matters for which percentage of programs, but here are some anecdotes of what can happen when not caring about NaN and rounding. We had a bug once which was really hard to track down. Turned out a simple lack of initialization of a variable in a simple piece of code for a running average: the programmer reasoned in the first iteration this number is going to be multiplied with 0.0 anyway, so no need to initialize it. However because there are many NaN representations, once in a while the uninitialized memory of the variable happened to represent NaN, and NaN * 0.0 is still NaN, and that would keep on propagating, oops. Likewise for rounding: we had some code calculating pixel positions and sometimes stuff was one pixel off resulting in weird visual artefacts. Turns out that when feeding the (faulty) rounding algorithm with results of some floating point calculations wouldn't do the expected thing but round down instead of up or so. Oops.

MrQuincle 9y ago

I was running MCMC for a while just in probability space. That didn't last long! It's tremendously useful to work in log probability space. It adds tremendous expressivity close to zero plus makes most multiplications sums. I think similar methods are probably used in other sectors. What I found remarkable though was the big difference I saw in old versus new compilers with respect to the diversity of random numbers generated. I first thought that it was the entropy available on my system but that didn't turn out to be the key factor. So watch out for old compilers!

s__mac 9y ago

The bummer about this to me is that the c99 guys took it upon themselves to extend IEEE semantics to complex numbers. To do that, you have to decide on a representation for infinity in complex numbers (there's really only one mathematically). They choose isinf(re(x)) || isinf(im(x)), so among other things, Inf + Nan*i is considered infinite...ok. And then they tried to make things consistent between real and complex numbers with this (made up) notion of complex infinity. The result being that there's edge cases they have to back up and 'fix' to be consistent. However, they could have just decided that complex arithmetic semantics follow naturally from floating point and whatever results ensue. So in the end this was all self-inflicted.