7 ms·

Much of the problem with floating point is that we intuitively expect it to follow the math rules we're familiar with, but they seem to, but often don't. Most

Much of the problem with floating point is that we intuitively expect it to follow the math rules we're familiar with, but they seem to, but often don't. Most programmers realize they are 'approximate' but they don't notice consequences, such as checking for exactly equal to zero, when you need to check for "close enough" depending on your context. More subtly, but perhaps more damaging, they break properties like associative and commutative. Different orders change values, you can add tiny values repeatedly to a large value, and it won't change, breaking testing of the value... The presentation goes over many of these, but the fundamental problem, to me, is that we're falsely presenting an interface suggesting "this acts like a number" when it doesn't in many key ways. We should perhaps call them approximate pseudo numbers or something. Don't allow operators, require functions like SortOfMaybeAdd, AcculumateList... Perhaps such measures will communicate better. A fascinating world we program in...

lmm 12y ago
I've long wanted a language that defaults to a decimal type (or perhaps forces the user to specify a precision), and provides hardware floating point only as an option where requested. Something like the way Python handles integers. Most of what C-family languages do has the wrong semantics. Any program which is happy to accept round-to-even and arbitrary increases in precision would probably accept the rearrangement of arithmetic under the assumption that it was commutative, but in e.g. Java there's no way to change rounding mode, and also no way to just request an approximate answer. The defaults are a compromise that satisfies no-one.
- dragonwriter 12y ago
  > I've long wanted a language that defaults to a decimal type (or perhaps forces the user to specify a precision), and provides hardware floating point only as an option where requested. Scheme (via its numeric tower) and some other language default to exact types (decimal or in some cases rational if they don't have a separate exact decimal type) and only use approximate floating point types if explicitly requested (including using by using a function which inherently produces an approximate output.) Haskell can be configured by way of the default specifier to do something analogous to this in its handling of ambiguous numeric types (which includes literals, which start out with a generic type of Num a => a) , though the default default uses double precision floating point for types that can't be constrained to arbitrary precision integers -- default (Integer, Double) -- so to do this you need something like default (Integer, Rational, Double.)
  - carterschonwald 12y ago
    In haskell, every literal is constructed via an Integer (big int) or Rational, which is then converted into the final representation. (roughly). I'm actually slowly doing some auditing of GHC Haskell's floating point machinery as I find the time.
    - dragonwriter 12y ago
      That makes sense as the underlying implementation approach, but I was more referring to the "final" representation (I've also edited the description as it regards Haskell with more details, since its not really about literal handling specifically, though that's subsumed within it.)
- ori_b 12y ago
  The problem is that decimal types don't make things any better, at least when you have finite precision. They have the same problems with rounding error. The error analysis is exactly the same when designing your algorithms, only in a different base. The only thing that they make easier is parsing and displaying numbers. The number of times where you deal with things that are guaranteed to be perfectly represented by powers of ten are relatively few.
  - acqq 12y ago
    > The number of times where you deal with things that are guaranteed to be perfectly represented by powers of ten are relatively few. For a physicist, a few. For an accountant, all.
wolfgke 12y ago
> More subtly, but perhaps more damaging, they break properties like associative and commutative. No: commutativity of multiplication and addition still holds, but associativity indeed does not. That associativity does not hold is a much more unpleasant than if associativity would hold but commutativity wouldn't (the mathematics of common associative, but non-commutative structures is understood quite well, say (non-abelian) groups and monoids).
- ColinDabritz 12y ago
  Ah yep. I was mixing things up a bit. The one that surprised me was that, in some cases: Large + small + small + small =! small + small + small + = Large Which is indeed associativity. Really underscores how complex things are.
- richardwhiuk 12y ago
  The other typically expected property that fails is distributivity.
  - ColinDabritz 12y ago
    So for floats: a ( b + c ) != a * b + a * c (in some cases)
- zwegner 12y ago
  Actually commutativity doesn't hold either, because of dealing with NaNs (at least on x86, on a quick search I can't find a reference for whether this behavior is part of IEEE 754 or just implementation defined). Operations with two NaNs will give the first operand. You can see this with a quick test program: #include <stdio.h> int main() { unsigned long long bits = 0x7FFC000000000000; double x = *(double *)&bits; bits += 1; double y = *(double *)&bits; double z = x + y, w = y + x; printf("%llx %llx\n", *(unsigned long long *)&z, *(unsigned long long *)&w); } Of course, this probably doesn't matter for most applications.
  - stephencanon 12y ago
dragonwriter 12y ago
> We should perhaps call them approximate pseudo numbers or something. More importantly, we just shouldn't make them the default types used for decimal literals (or, a fortiori, for all numbers in your language -- yes, JavaScript, I'm looking at you.) Premature optimization is the root of all kinds of evil -- or at least errors -- in programming, and "treat decimal literals as floating point because that's the most processor-friendly way of dealing with non-integer numbers" is the most pervasive premature optimization in computing.
- brucedawson 12y ago
  What alternative are you suggesting? Any finite precision numbers will have most of the same problems as IEEE floats. Decimal numbers are a specific case -- assuming them by default would be worse in some ways.
Tloewald 12y ago
"Much of the problem with floating point is that we intuitively expect it to follow the math rules we're familiar with, but they seem to, but often don't." Well... maybe. I don't think most people have the foggiest idea what floating point numbers are, or what the Real numbers they approximate are, so the idea that people (a) have a solid understanding of Math (let's just say Real Analysis) and that (b) this solid understanding is violated by the behavior of floating point numbers is a bit of a stretch. E.g. I doubt many programmers realize that 1/10 doesn't have a finite binary expansion.

lmm 12y ago

I've long wanted a language that defaults to a decimal type (or perhaps forces the user to specify a precision), and provides hardware floating point only as an option where requested. Something like the way Python handles integers. Most of what C-family languages do has the wrong semantics. Any program which is happy to accept round-to-even and arbitrary increases in precision would probably accept the rearrangement of arithmetic under the assumption that it was commutative, but in e.g. Java there's no way to change rounding mode, and also no way to just request an approximate answer. The defaults are a compromise that satisfies no-one.

dragonwriter 12y ago
> I've long wanted a language that defaults to a decimal type (or perhaps forces the user to specify a precision), and provides hardware floating point only as an option where requested. Scheme (via its numeric tower) and some other language default to exact types (decimal or in some cases rational if they don't have a separate exact decimal type) and only use approximate floating point types if explicitly requested (including using by using a function which inherently produces an approximate output.) Haskell can be configured by way of the default specifier to do something analogous to this in its handling of ambiguous numeric types (which includes literals, which start out with a generic type of Num a => a) , though the default default uses double precision floating point for types that can't be constrained to arbitrary precision integers -- default (Integer, Double) -- so to do this you need something like default (Integer, Rational, Double.)
- carterschonwald 12y ago
  In haskell, every literal is constructed via an Integer (big int) or Rational, which is then converted into the final representation. (roughly). I'm actually slowly doing some auditing of GHC Haskell's floating point machinery as I find the time.
  - dragonwriter 12y ago
    That makes sense as the underlying implementation approach, but I was more referring to the "final" representation (I've also edited the description as it regards Haskell with more details, since its not really about literal handling specifically, though that's subsumed within it.)
ori_b 12y ago
The problem is that decimal types don't make things any better, at least when you have finite precision. They have the same problems with rounding error. The error analysis is exactly the same when designing your algorithms, only in a different base. The only thing that they make easier is parsing and displaying numbers. The number of times where you deal with things that are guaranteed to be perfectly represented by powers of ten are relatively few.
- acqq 12y ago
  > The number of times where you deal with things that are guaranteed to be perfectly represented by powers of ten are relatively few. For a physicist, a few. For an accountant, all.

wolfgke 12y ago

> More subtly, but perhaps more damaging, they break properties like associative and commutative. No: commutativity of multiplication and addition still holds, but associativity indeed does not. That associativity does not hold is a much more unpleasant than if associativity would hold but commutativity wouldn't (the mathematics of common associative, but non-commutative structures is understood quite well, say (non-abelian) groups and monoids).

ColinDabritz 12y ago
Ah yep. I was mixing things up a bit. The one that surprised me was that, in some cases: Large + small + small + small =! small + small + small + = Large Which is indeed associativity. Really underscores how complex things are.
richardwhiuk 12y ago
The other typically expected property that fails is distributivity.
- ColinDabritz 12y ago
  So for floats: a ( b + c ) != a * b + a * c (in some cases)
zwegner 12y ago
Actually commutativity doesn't hold either, because of dealing with NaNs (at least on x86, on a quick search I can't find a reference for whether this behavior is part of IEEE 754 or just implementation defined). Operations with two NaNs will give the first operand. You can see this with a quick test program: #include <stdio.h> int main() { unsigned long long bits = 0x7FFC000000000000; double x = *(double *)&bits; bits += 1; double y = *(double *)&bits; double z = x + y, w = y + x; printf("%llx %llx\n", *(unsigned long long *)&z, *(unsigned long long *)&w); } Of course, this probably doesn't matter for most applications.
- stephencanon 12y ago

dragonwriter 12y ago

> We should perhaps call them approximate pseudo numbers or something. More importantly, we just shouldn't make them the default types used for decimal literals (or, a fortiori, for all numbers in your language -- yes, JavaScript, I'm looking at you.) Premature optimization is the root of all kinds of evil -- or at least errors -- in programming, and "treat decimal literals as floating point because that's the most processor-friendly way of dealing with non-integer numbers" is the most pervasive premature optimization in computing.

brucedawson 12y ago
What alternative are you suggesting? Any finite precision numbers will have most of the same problems as IEEE floats. Decimal numbers are a specific case -- assuming them by default would be worse in some ways.

Tloewald 12y ago

"Much of the problem with floating point is that we intuitively expect it to follow the math rules we're familiar with, but they seem to, but often don't." Well... maybe. I don't think most people have the foggiest idea what floating point numbers are, or what the Real numbers they approximate are, so the idea that people (a) have a solid understanding of Math (let's just say Real Analysis) and that (b) this solid understanding is violated by the behavior of floating point numbers is a bit of a stretch. E.g. I doubt many programmers realize that 1/10 doesn't have a finite binary expansion.