8 ms·

Yeah, the limitations of FP are well-known to anyone who does much numerical work. Floating point numbers are the optimal minimum message length method of repr

Yeah, the limitations of FP are well-known to anyone who does much numerical work. Floating point numbers are the optimal minimum message length method of representing reals with an improper Jeffery's prior distribution. A Jeffery's prior is a prior that is invariant under reparameterization, which is a mandatory property for approximating the reals. In this case, it is where Prob(log(|x|)) is proportional to a constant. Thus, we aren't going to ever do better than floats if we are programming on physical computers that exist in this universe. There is a reason why all numerical code uses them. Best to learn their limitations if you are going to use them, otherwise use arbitrary precision.

erik_seaberg 8y ago
There's no reason for every step of a computation to be confined to the same very small message length. And the necessary error analysis should be built into the language, preferably in the same "advanced users only, here be dragons" package as the imprecise types themselves.
- TFortunato 8y ago
  So interestingly, processor makers are on the same page with you re: computations, and lots of processors can internally do computations in "extended precision", e.g. 80-bit floats, only converting to/from 64-bit doubles at the start and end of the computation. https://en.m.wikipedia.org/wiki/Extended_precision https://en.m.wikipedia.org/wiki/Extended_precision
  - garmaine 8y ago
    That hasn’t been the case for over a decade.
    - TFortunato 8y ago
      How do you mean? The x86-64 instruction set / abi specifies long doubles as 80-bits, and still supports them ...
      - leiroigh 8y ago
        And nobody uses this terrible mis-feature in practice, everything runs via 64 bit xmm registers. Rightly so, because programmers want their optimizing compiler to decide when to put a variable on the stack and when to elide a store/load cycle by keeping it in a register. With 80 bit precision, this makes a semantic difference and you end up in volatile hell.
        TFortunato 8y ago
        Yeah I agree that everything typically runs in XMM registers and that's what people want. I'm not sure what about the availability of extended precision makes it s a misfeature? For some cases it IS what you want, and it's nice to be able to opt in to using it.. EDIT: If I had some application where I needed the extended range, like maybe I was going to run into the exact numbers above, I'd appreciate the ability to opt-in to this. Totally agree I wouldn't want the compiler to surprise me with it, but also not terrible, or useless. Code w/ Assembly: https://godbolt.org/z/W3ZmqJ https://godbolt.org/z/W3ZmqJ Output: https://onlinegdb.com/Sy_I3Q1ME https://onlinegdb.com/Sy_I3Q1ME
        BeeOnRope 8y ago
        To be fair, the problem you describe isn't inherent to 80-bit floating point values. If you use 80-bit values in your ABI or language definition, it won't occur - it occurs when you try to user a wider type to implement a narrower type, e.g., implementing 64-bit floats (as specified in the ABI or language) with 80-bit operations. In that case, the extra precision is present and "carried across" operation when registers or the dedicated floating point stack is used, but is discarded when values are stored to a narrower 64-bit location. So the problem is one really of mismatch between the language/ABI size and the supported hardware size. Of course, 80 bits isn't a popular floating point size any more in modern languages, so this happens a lot.
      - loeg 8y ago
        The x86-64 instruction set does not specify any particular ABI and definitely does not specify the precision of a C language type.
    - ealloc 8y ago
      It may be coming back though. IBM's new and rising supercomputer architecture, POWER9, supports hardware IEEE binary128 floats (quad precision). Their press claims the current fastest supercomputer in the world uses POWER9. The ppc64 architecture (still produced by IBM) supports "double-double" precision for the long-double type, which is a bit hacky and software-defined, but has 106 bit mantissa. And ARM's aarch64 architecture supports IEEE binary128 long-doubles as well, though it is implemented in software now (by compiler). Maybe they plan a hardware implementation in the future?
  - deleted 8y ago
    [deleted]
- lenticular 8y ago
  A small message length means small memory, which is important in physical computers. It is information-theoretic optimal. This is a well-defined term.
- deleted 8y ago
  [deleted]
3pt14159 8y ago
Outside of the academic world decimals are almost always a better solution if performance isn't critical. Most logic is multiplicative. For example, apply a 30% tax on a dollar quantity and display both subtotal and grand total. With floats, there are inequalities. With decimal there usually aren't unless you're dividing, but we already have to deal with divide errors in base ten, and it is much more likely to need to represent 0.30 than 1/3 and because decimal shares a base with binary (since it's factors are 5 and 2) binary doesn't really get us anything but headaches anyway. It's true that there are still gotchas, but they happen less often and usually don't end up looking stupid and weird for no reason. That 0.1 + 0.2 = 0.300000000000001 is dumb and we all know it.
- ChrisLomont 8y ago
  You cannot do anything in finance with such reasoning. Take something simple, say a mortgage at 5% compounded 12 times a year. To compute payments using some fixed length representation or decimal is going to lead to more error than to use the usual floating point. This rabbit hole would continue for many applications. Floating point makes them all much easier to do well.
  - jes5199 8y ago
    Have you worked on finance software? I have - we always used ints for everything, so we could avoid rounding suprises
    - bigiain 8y ago
      I did a web project in the gambling space ~10 years back - we were legally required to perform all calculations as integers in ten thousandths of a cent (or millionths of a dollar). We chose to _not_ do _any_ calculations client side in Javascript...
      - dotancohen 8y ago
        Which regulation is that? I've worked on financial applications, but not gambling, and I've not heard of this regulation. I should probably know about it!
vanderZwan 8y ago
> Thus, we aren't going to ever do better than floats if we are programming on physical computers that exist in this universe. Maybe, but that doesn't mean the particular implementation of floats being used is the best one. See also: Unums and Posits https://posithub.org/ https://posithub.org/
- nestorD 8y ago
  To my understanding, experiments with Unums showed shortcoming that Gustafson didn't anticipate and lead to Posits which drop the fixed length constraint. Doing that makes improving precision a lot easier but at the cost of computation time. Overall I am not convinced that the current implementation is optimal but it is a very good trade-off between speed and precision.
  - vanderZwan 8y ago
    > Doing that makes improving precision a lot easier but at the cost of computation time. Not quite. The difference in computation time is the current the lack of hardware support, not something inherent to the underlying encoding method. So in practice you are right, but in, for example, embedded contexts without floating point hardware, the performance advantages of IEEE floats should disappear (especially if using a 16 or 8 bit posit suffices). Posits are simpler to implement than IEEE floats (less edge cases) and use more bits for actual numbers whereas IEEE floats waste about half on NaNs. The use of tapered precision is also nice.
    - tntn 8y ago
      Even if hardware support existed, it seems like a variable length encoding has some inherent overhead relative to a fixed length encoding. If you have a "base length" of e.g. 32 bits and occasionally expand to 64, there's an inherent cost there in both computation and memory, presumably for greater precision. Perhaps that overhead could be minimal with hardware support, but it seems it must have some.
      - 8y ago
erpellan 8y ago
The limitations should be well known. One of the first things I check when joining a financial software project is how the system represents money. I’m rarely surprised. (It’s inevitably floats or doubles)
ayidnelm 8y ago
> Floating point numbers are the optimal minimum message length method of representing reals with an improper Jeffery's prior distribution. Do you have a link to a proof or discussion of this? I haven't heard this before and I would love to have this statement unpacked a little more.

erik_seaberg 8y ago

There's no reason for every step of a computation to be confined to the same very small message length. And the necessary error analysis should be built into the language, preferably in the same "advanced users only, here be dragons" package as the imprecise types themselves.

TFortunato 8y ago
So interestingly, processor makers are on the same page with you re: computations, and lots of processors can internally do computations in "extended precision", e.g. 80-bit floats, only converting to/from 64-bit doubles at the start and end of the computation. https://en.m.wikipedia.org/wiki/Extended_precision https://en.m.wikipedia.org/wiki/Extended_precision
- garmaine 8y ago
  That hasn’t been the case for over a decade.
  - TFortunato 8y ago
    How do you mean? The x86-64 instruction set / abi specifies long doubles as 80-bits, and still supports them ...
    - leiroigh 8y ago
      And nobody uses this terrible mis-feature in practice, everything runs via 64 bit xmm registers. Rightly so, because programmers want their optimizing compiler to decide when to put a variable on the stack and when to elide a store/load cycle by keeping it in a register. With 80 bit precision, this makes a semantic difference and you end up in volatile hell.
      - TFortunato 8y ago
        Yeah I agree that everything typically runs in XMM registers and that's what people want. I'm not sure what about the availability of extended precision makes it s a misfeature? For some cases it IS what you want, and it's nice to be able to opt in to using it.. EDIT: If I had some application where I needed the extended range, like maybe I was going to run into the exact numbers above, I'd appreciate the ability to opt-in to this. Totally agree I wouldn't want the compiler to surprise me with it, but also not terrible, or useless. Code w/ Assembly: https://godbolt.org/z/W3ZmqJ https://godbolt.org/z/W3ZmqJ Output: https://onlinegdb.com/Sy_I3Q1ME https://onlinegdb.com/Sy_I3Q1ME
      - BeeOnRope 8y ago
        To be fair, the problem you describe isn't inherent to 80-bit floating point values. If you use 80-bit values in your ABI or language definition, it won't occur - it occurs when you try to user a wider type to implement a narrower type, e.g., implementing 64-bit floats (as specified in the ABI or language) with 80-bit operations. In that case, the extra precision is present and "carried across" operation when registers or the dedicated floating point stack is used, but is discarded when values are stored to a narrower 64-bit location. So the problem is one really of mismatch between the language/ABI size and the supported hardware size. Of course, 80 bits isn't a popular floating point size any more in modern languages, so this happens a lot.
    - loeg 8y ago
      The x86-64 instruction set does not specify any particular ABI and definitely does not specify the precision of a C language type.
  - ealloc 8y ago
    It may be coming back though. IBM's new and rising supercomputer architecture, POWER9, supports hardware IEEE binary128 floats (quad precision). Their press claims the current fastest supercomputer in the world uses POWER9. The ppc64 architecture (still produced by IBM) supports "double-double" precision for the long-double type, which is a bit hacky and software-defined, but has 106 bit mantissa. And ARM's aarch64 architecture supports IEEE binary128 long-doubles as well, though it is implemented in software now (by compiler). Maybe they plan a hardware implementation in the future?
- deleted 8y ago
  [deleted]
lenticular 8y ago
A small message length means small memory, which is important in physical computers. It is information-theoretic optimal. This is a well-defined term.
deleted 8y ago
[deleted]

3pt14159 8y ago

Outside of the academic world decimals are almost always a better solution if performance isn't critical. Most logic is multiplicative. For example, apply a 30% tax on a dollar quantity and display both subtotal and grand total. With floats, there are inequalities. With decimal there usually aren't unless you're dividing, but we already have to deal with divide errors in base ten, and it is much more likely to need to represent 0.30 than 1/3 and because decimal shares a base with binary (since it's factors are 5 and 2) binary doesn't really get us anything but headaches anyway. It's true that there are still gotchas, but they happen less often and usually don't end up looking stupid and weird for no reason. That 0.1 + 0.2 = 0.300000000000001 is dumb and we all know it.

ChrisLomont 8y ago
You cannot do anything in finance with such reasoning. Take something simple, say a mortgage at 5% compounded 12 times a year. To compute payments using some fixed length representation or decimal is going to lead to more error than to use the usual floating point. This rabbit hole would continue for many applications. Floating point makes them all much easier to do well.
- jes5199 8y ago
  Have you worked on finance software? I have - we always used ints for everything, so we could avoid rounding suprises
  - bigiain 8y ago
    I did a web project in the gambling space ~10 years back - we were legally required to perform all calculations as integers in ten thousandths of a cent (or millionths of a dollar). We chose to _not_ do _any_ calculations client side in Javascript...
    - dotancohen 8y ago
      Which regulation is that? I've worked on financial applications, but not gambling, and I've not heard of this regulation. I should probably know about it!

vanderZwan 8y ago

> Thus, we aren't going to ever do better than floats if we are programming on physical computers that exist in this universe. Maybe, but that doesn't mean the particular implementation of floats being used is the best one. See also: Unums and Posits https://posithub.org/ https://posithub.org/

nestorD 8y ago
To my understanding, experiments with Unums showed shortcoming that Gustafson didn't anticipate and lead to Posits which drop the fixed length constraint. Doing that makes improving precision a lot easier but at the cost of computation time. Overall I am not convinced that the current implementation is optimal but it is a very good trade-off between speed and precision.
- vanderZwan 8y ago
  > Doing that makes improving precision a lot easier but at the cost of computation time. Not quite. The difference in computation time is the current the lack of hardware support, not something inherent to the underlying encoding method. So in practice you are right, but in, for example, embedded contexts without floating point hardware, the performance advantages of IEEE floats should disappear (especially if using a 16 or 8 bit posit suffices). Posits are simpler to implement than IEEE floats (less edge cases) and use more bits for actual numbers whereas IEEE floats waste about half on NaNs. The use of tapered precision is also nice.
  - tntn 8y ago
    Even if hardware support existed, it seems like a variable length encoding has some inherent overhead relative to a fixed length encoding. If you have a "base length" of e.g. 32 bits and occasionally expand to 64, there's an inherent cost there in both computation and memory, presumably for greater precision. Perhaps that overhead could be minimal with hardware support, but it seems it must have some.
    - 8y ago

erpellan 8y ago

The limitations should be well known. One of the first things I check when joining a financial software project is how the system represents money. I’m rarely surprised. (It’s inevitably floats or doubles)

ayidnelm 8y ago

> Floating point numbers are the optimal minimum message length method of representing reals with an improper Jeffery's prior distribution. Do you have a link to a proof or discussion of this? I haven't heard this before and I would love to have this statement unpacked a little more.