6 ms·

Hoo boy. This is what happens the first time C programmers start to work with floating point and don't know the fundamentals. When you work with floating point

Hoo boy. This is what happens the first time C programmers start to work with floating point and don't know the fundamentals. When you work with floating point, you need to remember you work with a tolerance to epsilon for comparisons because you are rounding to 1/n^2 precision and different floating point units perform the conversion in different ways. You must abandon the idea of '==' for floats. This is why his code is unpredictable, because you cannot guarantee the conversion of any integer to and from float is the same number. Period. The LSBs of the mantissa can and do change, which is why we mask to a precision or use signal-to-noise comparisons when evaluation bit drift between FP computations. He has the first part correct, < and > are your friend with FP. But to get past the '==' hurdle, he needs to define his tolerance, the code should be something like: if (fabs(f1 - f2) > TOLERANCE) ... fits = true. I was irked by his arrogance when he asks, "Intel CPUs have a history of bugs. Did I hit one of those?" First, learn about floating point, then, work on an FPUnit team for 10 years, and even then, don't assume you're smarter than a team of floating point architects, you're not.

chungus_khan 7y ago
Absolutely. It's true that Intel CPUs have a history of bugs, and it's always a danger when working with any CPU. But unless you really, really know what you're doing you probably didn't, especially if you are doing something relatively basic and think you've somehow uncovered a new bug.
thanatropism 7y ago
How is == still defined for floats (in typeful languages)? Fixed precision decimals should be on offer and suggested by compilers in response to an equality-on-floats.
- ambrop7 7y ago
  Each floating point value that is not a NaN represents a certain real number, -inf or +inf (this can be expressed in terms of the sign bit, exponent and mantissa). Knowing that, a == b when neither operand is a NaN is defined as equality of what they represent, in purely mathematical terms. Similar can be said for inequality operators. Be aware that +0.0 and -0.0 are different floating point values but represent the same real number, so +0.0 == -0.0 follows. People who say == means nothing for floating point and you always need epsilon checks are wrong, plain and simple. == is very well defined. Don't confuse the definition of floating point operations with common practices for using them effectively. You can iterate through all non-NaN values and check that successive ones are indeed not equal: #include <math.h> #include <stdint.h> #include <assert.h> #include <stdio.h> #include <inttypes.h> int main() { float x = (float)-INFINITY; uint64_t count = 1; while (x != (float)INFINITY) { float y = nextafterf(x, (float)INFINITY); assert(y != x); x = y; ++count; } printf("Found %" PRIu64 " floats.\n", count); return 0; } $ gcc -std=c99 -O3 a.c -lm -o a $ ./a Found 4278190081 floats. (a little bit harder for doubles) Interestingly, this only finds one zero (-0.0), hence the assert doesn't actually fail around zero.
  - thanatropism 7y ago
    Each int also corresponds to a certain real number, but if you write "x*(1/x)==1" for integer x, you'll either (a) get x casted to a float (b) get `div` instead (like in Python 2) and obtain a false result (c) get cursed out with a type error. These three options are available because it's possible to determine whether a given real number is representable as an int or not. This is not possible with floats.
    - ambrop7 7y ago
      Programming languages aren't able to express arbitrary real numbers, so to "determine whether a given real number is representable as an int or not" is mostly meaningless in a programming language as opposed to a computer algebra system. What you can do is: - determine if a float is an integer (trunc(x) == x), - convert a float to a certain integer type with some kind of rounding, or get an error if it's out of range (see my comment with double_to_uint64), - convert a float to a certain integer type exactly, or get an error if it's not representable (e.g. by doing both of the above). The basic reason that so many people fail to use floats correctly is that they act like operations on floats are equivalent to operations on the real numbers they represent, when in fact they are usually defined as the operation on real numbers rounded to a representable value.
      - thanatropism 7y ago
        > Programming languages aren't able to express arbitrary real numbers, No system of finite representations is able to express arbitrary real numbers.
        quickthrower2 7y ago
        Depends what you mean by an arbitrary real number. Any constructed real number could be stored as it’s construction steps. Eg use a series for Pi. But paradoxically you can store “any” real number, eg the number made up of all future lottery draws.
titzer 7y ago
Floating point is indeed hard. It's made harder because the C programming language, due to its history supporting a large number of targets, especially for high performance computing, does not even mandate IEEE 754. (C predated IEEE 754). IEEE 754 is actually very precise about rounding, rounding modes, etc. The x87 floating point coprocessor is also heavily to blame, because of its internal 80-bit precision. Decades of headaches. Other headaches come from ARM NEON's vector instructions not implementing subnormals (RTZ mode), which is not IEEE, which makes vector math differ from scalar math in corner cases. GPUs also went through a similar evolution. Slowly the industry is moving to all-IEEE 754 compliant arithmetic. There's a lot more to say, but yes, I agree with you, it's complicated.
- Gibbon1 7y ago
  Yeah and then we have the problem that most floating point variables shouldn't be IEEE 754. And we're mostly fucked because most languages don't support anything else.
  - mark-r 7y ago
    If they shouldn't be IEEE 754 then what would you use instead? IEEE 754 exists because a lot of smart people put a lot of thought into what the best floating point representation should be. Then the hardware folks put a lot of work into making it fast and accurate.
    - Gibbon1 7y ago
      IEEE 754 is designed for scientific computation and simulation. And it's very good for that. However it's terrible for representing decimal numbers and fractions. You end up with the exact problems described in the article. It's kind of a problem since most programmers aren't writing programs to do scientific/modeling. They're writing programs that are doing decimal math and currency where IEEE 754 is inappropriate.
      - mark-r 7y ago
slavik81 7y ago
Choosing a tolerance is hard enough when you know the floating point model you're using, but it seems like an impossible task to try to support all possible floating point hardware. I couldn't tell you what guarantees you have when __STDC_IEC_559__ remains undefined. These days I write my floating point code assuming float is a 32-bit IEEE 754 floating point number, and double is 64-bit. You can get those guarantees on any desktop hardware with the right flags, and the same semantics are commonly available on other platforms too. Picking a well-defined implementation makes it much easier to reason about conversions between integer and floating point types. In fact, it allows you to reason about a lot of operations, e.g. 1.0 + 2.0 == 3.0 is true; (float)183 == 183.0 is true; 0.1 == (double)0.1f is false; etc.
mark-r 7y ago
> you cannot guarantee the conversion of any integer to and from float is the same number. Totally false. Any integer between -(2^53) and 2^53 can be converted to IEEE 64-bit double without any loss of information.
- coldtea 7y ago
  Read the parent again. Those are a specific integer range, not "any integer".
  - mark-r 7y ago
    I'm just saying that the claim made by the parent is overly strong. For most of the integers that we will deal with, they can be converted back and forth to doubles without problems. Even when they can't the double will contain an integer value, it just won't be the correct value. The usual problems people have with doubles are the bits to the right of the decimal point.
ambrop7 7y ago
You don't always need epsilon. I've written lots of floating point code that works well without epsilon checks. See my comment in this thread (double_to_uint64) how what the OP needs can be done correctly without an epsilon check.
- 0xff00ffee 7y ago
  Calm down. You're smart, we get it. You've misquoted my comment twice, please stop.

chungus_khan 7y ago

Absolutely. It's true that Intel CPUs have a history of bugs, and it's always a danger when working with any CPU. But unless you really, really know what you're doing you probably didn't, especially if you are doing something relatively basic and think you've somehow uncovered a new bug.

thanatropism 7y ago

How is == still defined for floats (in typeful languages)? Fixed precision decimals should be on offer and suggested by compilers in response to an equality-on-floats.

ambrop7 7y ago
Each floating point value that is not a NaN represents a certain real number, -inf or +inf (this can be expressed in terms of the sign bit, exponent and mantissa). Knowing that, a == b when neither operand is a NaN is defined as equality of what they represent, in purely mathematical terms. Similar can be said for inequality operators. Be aware that +0.0 and -0.0 are different floating point values but represent the same real number, so +0.0 == -0.0 follows. People who say == means nothing for floating point and you always need epsilon checks are wrong, plain and simple. == is very well defined. Don't confuse the definition of floating point operations with common practices for using them effectively. You can iterate through all non-NaN values and check that successive ones are indeed not equal: #include <math.h> #include <stdint.h> #include <assert.h> #include <stdio.h> #include <inttypes.h> int main() { float x = (float)-INFINITY; uint64_t count = 1; while (x != (float)INFINITY) { float y = nextafterf(x, (float)INFINITY); assert(y != x); x = y; ++count; } printf("Found %" PRIu64 " floats.\n", count); return 0; } $ gcc -std=c99 -O3 a.c -lm -o a $ ./a Found 4278190081 floats. (a little bit harder for doubles) Interestingly, this only finds one zero (-0.0), hence the assert doesn't actually fail around zero.
- thanatropism 7y ago
  Each int also corresponds to a certain real number, but if you write "x*(1/x)==1" for integer x, you'll either (a) get x casted to a float (b) get `div` instead (like in Python 2) and obtain a false result (c) get cursed out with a type error. These three options are available because it's possible to determine whether a given real number is representable as an int or not. This is not possible with floats.
  - ambrop7 7y ago
    Programming languages aren't able to express arbitrary real numbers, so to "determine whether a given real number is representable as an int or not" is mostly meaningless in a programming language as opposed to a computer algebra system. What you can do is: - determine if a float is an integer (trunc(x) == x), - convert a float to a certain integer type with some kind of rounding, or get an error if it's out of range (see my comment with double_to_uint64), - convert a float to a certain integer type exactly, or get an error if it's not representable (e.g. by doing both of the above). The basic reason that so many people fail to use floats correctly is that they act like operations on floats are equivalent to operations on the real numbers they represent, when in fact they are usually defined as the operation on real numbers rounded to a representable value.
    - thanatropism 7y ago
      > Programming languages aren't able to express arbitrary real numbers, No system of finite representations is able to express arbitrary real numbers.
      - quickthrower2 7y ago
        Depends what you mean by an arbitrary real number. Any constructed real number could be stored as it’s construction steps. Eg use a series for Pi. But paradoxically you can store “any” real number, eg the number made up of all future lottery draws.

titzer 7y ago

Floating point is indeed hard. It's made harder because the C programming language, due to its history supporting a large number of targets, especially for high performance computing, does not even mandate IEEE 754. (C predated IEEE 754). IEEE 754 is actually very precise about rounding, rounding modes, etc. The x87 floating point coprocessor is also heavily to blame, because of its internal 80-bit precision. Decades of headaches. Other headaches come from ARM NEON's vector instructions not implementing subnormals (RTZ mode), which is not IEEE, which makes vector math differ from scalar math in corner cases. GPUs also went through a similar evolution. Slowly the industry is moving to all-IEEE 754 compliant arithmetic. There's a lot more to say, but yes, I agree with you, it's complicated.

Gibbon1 7y ago
Yeah and then we have the problem that most floating point variables shouldn't be IEEE 754. And we're mostly fucked because most languages don't support anything else.
- mark-r 7y ago
  If they shouldn't be IEEE 754 then what would you use instead? IEEE 754 exists because a lot of smart people put a lot of thought into what the best floating point representation should be. Then the hardware folks put a lot of work into making it fast and accurate.
  - Gibbon1 7y ago
    IEEE 754 is designed for scientific computation and simulation. And it's very good for that. However it's terrible for representing decimal numbers and fractions. You end up with the exact problems described in the article. It's kind of a problem since most programmers aren't writing programs to do scientific/modeling. They're writing programs that are doing decimal math and currency where IEEE 754 is inappropriate.
    - mark-r 7y ago

slavik81 7y ago

Choosing a tolerance is hard enough when you know the floating point model you're using, but it seems like an impossible task to try to support all possible floating point hardware. I couldn't tell you what guarantees you have when __STDC_IEC_559__ remains undefined. These days I write my floating point code assuming float is a 32-bit IEEE 754 floating point number, and double is 64-bit. You can get those guarantees on any desktop hardware with the right flags, and the same semantics are commonly available on other platforms too. Picking a well-defined implementation makes it much easier to reason about conversions between integer and floating point types. In fact, it allows you to reason about a lot of operations, e.g. 1.0 + 2.0 == 3.0 is true; (float)183 == 183.0 is true; 0.1 == (double)0.1f is false; etc.

mark-r 7y ago

> you cannot guarantee the conversion of any integer to and from float is the same number. Totally false. Any integer between -(2^53) and 2^53 can be converted to IEEE 64-bit double without any loss of information.

coldtea 7y ago
Read the parent again. Those are a specific integer range, not "any integer".
- mark-r 7y ago
  I'm just saying that the claim made by the parent is overly strong. For most of the integers that we will deal with, they can be converted back and forth to doubles without problems. Even when they can't the double will contain an integer value, it just won't be the correct value. The usual problems people have with doubles are the bits to the right of the decimal point.

ambrop7 7y ago

You don't always need epsilon. I've written lots of floating point code that works well without epsilon checks. See my comment in this thread (double_to_uint64) how what the OP needs can be done correctly without an epsilon check.

0xff00ffee 7y ago
Calm down. You're smart, we get it. You've misquoted my comment twice, please stop.