4 ms·

Am I the only person who thinks of IEEE 754 as a completely well-defined data structure with well-defined arithmetic operations, where two numbers are equal iff

Am I the only person who thinks of IEEE 754 as a completely well-defined data structure with well-defined arithmetic operations, where two numbers are equal iff their bits are equal? Tiny discrepancies don't randomly appear during float operations, and two different black box systems written from the same order of operations spec sheet (given proper compiler strict math settings) should be able to output the exact same float value given the same input.

macrolocal 4y ago
Not at all, eg. Knuth's TAOCP, Vol. 2, Section 4.2.2 A.
adgjlsfhk1 4y ago
There are 2 problems with this. The first is that CPUs want to do out of order execution and vectorization, so not allowing reordering can come at a significant cost. The second problem is that different LIBMs and cpus will change the result of transcendental functions like exp.
Someone 4y ago
> where two numbers are equal iff their bits are equal? That’s not true. One of the features of IEEE floats is the existence of NaNs that aren’t (according to the standard) equal to themselves.
tomxor 4y ago
> Am I the only person who thinks of IEEE 754 as a completely well-defined data structure Not at all... but I think there are a large number of people who think they can imagine a better format and yet don't fully understand or appreciate much of the design of IEEE754, and they tend to be quite noisy. I think the lack of appreciation is ironically due to how damn good a format it is. It manages to hide the myriad deficiencies through such a good balance of compromises that it casts this illusion that perfect fp math is possible in a generalisable enough way. So on the occasions it surfaces the true reality, people assume it must be some kind of defect that is easily addressable with a new format; rather than an intentional decision to maximise correctness and utility by carefully choosing where and how to be incorrect. FP math is the most underestimated areas by programmers IMO, it's just taken for granted so much, which is probably a testament to how well IEEE754 works.
- constantcrying 4y ago
  You are absolutely right. So many times I have seen people complain about this or that "flaw" in floating point arithmetic and proposing this or that alternative system. It always makes me question whether they have actually tried using these alternatives... Giving people the illusion that they can work with real numbers on a computer by itself is pretty magical. But the fact that 32bits of data are not enough to hold information about all real numbers and that floats aren't actually real numbers should not be surprising. To me it seems most people don't even understand what floating point numbers are, if you know a little bit all the "weird" behavior starts to make sense and you understand why it has to be that way and how much thought and genius has been put into this datatype.
racingmars 4y ago
I'm not sure what you mean by this. It is true that two compliant systems will produce the same output for the same sequence of inputs, but that isn't the entire scope of when the concept of "equals" might matter. > Tiny discrepancies don't randomly appear during float operations They don't "randomly" appear, because as you say IEEE 754 is well-defined, but discrepancies _do_ appear, and if you care about the concept of equality, they certainly matter. Take a trivial example in C: #include <stdio.h> int main(void) { volatile double a, b, c, d, e; a = 5.0; b = a/3.0; c = b/3.0; d = c/3.0; e = d*27.0; if (a==e) printf("EQUAL\n"); else printf("NOT EQUAL\n"); return 0; } I think everyone would agree that 5 / 3 / 3 / 3 * 27 = 5, or in general that a / 3 / 3 / 3 * 27 = a. But this program, when executed on a system properly implementing IEEE 754, will report that a and e are not equal. Their bits are different. Hence why when you're talking about comparing equality of IEEE 754 values, you can't just do a comparison of the bits in storage. You're right that all (correctly implemented) systems will accumulate errors in the same way (assuming you've selected the same rounding mode, etc.), but I don't see how that negates the need to handle the fact that the representation in memory of e will not equal a even though they _should_ represent the same value. Given the sequence of operations performed, how much "slop" should you allow when comparing a to e? Certainly more than 0 which a bit-by-bit comparison implies.
- whycombinetor 4y ago
  It's no "error" (ha) that you can't exactly represent 5/3 in base 2 number system - it can only represent numbers of the form (-1)^a*2^b*c for integer a,b,c. If you are trying to perform repeated calculations that accumulate some arithmetic error at every step, then ieee754 probably isn't the best data type for the task at hand. Fractional calculators (which allow exact representation of fractions like 5/3 without being limited by base or precision) or "constructive reals" - or simply keeping track of the numerator and comparing if the sum of the numerators is equal to the denominator - would avoid such issues. As another reply to my comment said, it is about using the right data type for the task, and float is a pretty good compromise between many different needs but isn't appropriate for literally everything.

macrolocal 4y ago

Not at all, eg. Knuth's TAOCP, Vol. 2, Section 4.2.2 A.

adgjlsfhk1 4y ago

There are 2 problems with this. The first is that CPUs want to do out of order execution and vectorization, so not allowing reordering can come at a significant cost. The second problem is that different LIBMs and cpus will change the result of transcendental functions like exp.

Someone 4y ago

> where two numbers are equal iff their bits are equal? That’s not true. One of the features of IEEE floats is the existence of NaNs that aren’t (according to the standard) equal to themselves.

tomxor 4y ago

> Am I the only person who thinks of IEEE 754 as a completely well-defined data structure Not at all... but I think there are a large number of people who think they can imagine a better format and yet don't fully understand or appreciate much of the design of IEEE754, and they tend to be quite noisy. I think the lack of appreciation is ironically due to how damn good a format it is. It manages to hide the myriad deficiencies through such a good balance of compromises that it casts this illusion that perfect fp math is possible in a generalisable enough way. So on the occasions it surfaces the true reality, people assume it must be some kind of defect that is easily addressable with a new format; rather than an intentional decision to maximise correctness and utility by carefully choosing where and how to be incorrect. FP math is the most underestimated areas by programmers IMO, it's just taken for granted so much, which is probably a testament to how well IEEE754 works.

constantcrying 4y ago
You are absolutely right. So many times I have seen people complain about this or that "flaw" in floating point arithmetic and proposing this or that alternative system. It always makes me question whether they have actually tried using these alternatives... Giving people the illusion that they can work with real numbers on a computer by itself is pretty magical. But the fact that 32bits of data are not enough to hold information about all real numbers and that floats aren't actually real numbers should not be surprising. To me it seems most people don't even understand what floating point numbers are, if you know a little bit all the "weird" behavior starts to make sense and you understand why it has to be that way and how much thought and genius has been put into this datatype.

racingmars 4y ago

I'm not sure what you mean by this. It is true that two compliant systems will produce the same output for the same sequence of inputs, but that isn't the entire scope of when the concept of "equals" might matter. > Tiny discrepancies don't randomly appear during float operations They don't "randomly" appear, because as you say IEEE 754 is well-defined, but discrepancies _do_ appear, and if you care about the concept of equality, they certainly matter. Take a trivial example in C: #include <stdio.h> int main(void) { volatile double a, b, c, d, e; a = 5.0; b = a/3.0; c = b/3.0; d = c/3.0; e = d*27.0; if (a==e) printf("EQUAL\n"); else printf("NOT EQUAL\n"); return 0; } I think everyone would agree that 5 / 3 / 3 / 3 * 27 = 5, or in general that a / 3 / 3 / 3 * 27 = a. But this program, when executed on a system properly implementing IEEE 754, will report that a and e are not equal. Their bits are different. Hence why when you're talking about comparing equality of IEEE 754 values, you can't just do a comparison of the bits in storage. You're right that all (correctly implemented) systems will accumulate errors in the same way (assuming you've selected the same rounding mode, etc.), but I don't see how that negates the need to handle the fact that the representation in memory of e will not equal a even though they _should_ represent the same value. Given the sequence of operations performed, how much "slop" should you allow when comparing a to e? Certainly more than 0 which a bit-by-bit comparison implies.

whycombinetor 4y ago
It's no "error" (ha) that you can't exactly represent 5/3 in base 2 number system - it can only represent numbers of the form (-1)^a*2^b*c for integer a,b,c. If you are trying to perform repeated calculations that accumulate some arithmetic error at every step, then ieee754 probably isn't the best data type for the task at hand. Fractional calculators (which allow exact representation of fractions like 5/3 without being limited by base or precision) or "constructive reals" - or simply keeping track of the numerator and comparing if the sum of the numerators is equal to the denominator - would avoid such issues. As another reply to my comment said, it is about using the right data type for the task, and float is a pretty good compromise between many different needs but isn't appropriate for literally everything.