6 ms·

> considering the problem is to fit the reals into 64/32/16 bits and have fast math Floating-point numbers (and IEEE-754 in particular) are a good solution to

by svat 8y ago

> considering the problem is to fit the reals into 64/32/16 bits and have fast math Floating-point numbers (and IEEE-754 in particular) are a good solution to this problem, but is it the right problem? I think the "minimum of surprises" part isn't true. Many programmers develop incorrect mental models when starting to program, and get no feedback to correct them until much later (when they get surprised). It is true that for the problem you mentioned, IEEE 754 is a good tradeoff (though Gustafson has some interesting ideas with “unums”: https://web.stanford.edu/class/ee380/Abstracts/170201-slides.pdf https://web.stanford.edu/class/ee380/Abstracts/170201-slides... / http://johngustafson.net/unums.html http://johngustafson.net/unums.html / https://en.wikipedia.org/w/index.php?title=Unum_(number_format)&oldid=873507492 https://en.wikipedia.org/w/index.php?title=Unum_(number_form... ). But many programmers do not realize how they are approximating, and the "fixed number of bits" may not be a strict requirement in many cases. (For example, languages that have arbitrary precision integers by default don't seem to suffer for it overall, relative to those that have 32-bit or 64-bit integers.) Even without moving away from the IEEE-754 standard, there are ways languages could be designed to minimize surprises. A couple of crazy ideas: Imagine if typing the literal 0.1 into a program gave an error or warning saying it cannot be represented exactly and has been approximated to 0.100000000000000005551, and one had to type "~0.1" or "nearest(0.1)" or add something at the top of the program to suppress such errors/warnings. At a very slight cost, one gives more feedback to the user to either fix their mental model or switch to a more appropriate type for their application. Similarly if the default print/to-string on a float showed ranges (e.g. printing the single-precision float corresponding to 0.1, namely 0.100000001490116119385, would show "between 0.09999999776482582 and 0.10000000521540642" or whatever) and one had to do an extra step or add something to the top of the program to get the shortest approximation ("0.1").

crankylinuxuser 8y ago
When NASA can't even get it right, because of "surprises", there's no chance in hell I'm blaming us mere mortal programmers... or even 10x wizards. (0) It's time to look at other ways to depict fractional parts of numbers in a computer. I know that one can express any rational number as a integer fraction. And our computers are incapable of expressing a irrational number exactly - it does so to a certain precision... In other words, every number a computer expresses is a rational number. The exception is if the computer could express irrational numbers as symbolics, then we could work with the symbolic instead. And then as a last pass, the symbolic could convert to a imprecise rational depiction, or express as its native type. (0) https://itsfoss.com/a-floating-point-error-that-caused-a-damage-worth-half-a-billion/ https://itsfoss.com/a-floating-point-error-that-caused-a-dam...
- saagarjha 8y ago
  Rationals get unwieldy quickly, even with the simplest of arithmetic. A couple of additions is enough to get a large denominator.
  - bjourne 8y ago
    That has not been my experience using a language with builtin support for rationals. The rational is simplified after each operation so it never grows unwieldy large. It is slower than floats, but imo vastly superior for most use cases.
    - Retra 8y ago
      "Most" use cases? You mean the use cases that don't involve heavy math, like graphics, physics, and statistics?
    - johndough 8y ago
      > The rational is simplified after each operation so it never grows unwieldy large. We recently had an exponential memory growth bug because rationals can not always be simplified, for example if you start with (2/3) and repeatedly square it. Fortunately, this was not in user-facing code, so there was no chance of a denial-of-service attack, but that's definitely something to watch out for with rationals.
    - stephencanon 8y ago
      Even simplifying after every operation, in the typical case exact rationals grow exponentially in the number of terms in the computation. This means that either: (a) you cannot use them for any non-trivial computation. (b) you have to round them, in which case they are strictly worse than floating-point numbers because they have redundant representations and a very non-uniform distribution.
- empath75 8y ago
  You can work directly in reals. http://hackage.haskell.org/package/exact-real http://hackage.haskell.org/package/exact-real
deleted 8y ago
[deleted]
sampo 8y ago
There is a proposal about "unum" or "posit" number system. They give more precision for small numbers (small meaning smaller than about 10^70, for 64bit numbers), less precision for huge numbers, and an overall larger range, than the floating point system. https://en.wikipedia.org/wiki/Unum_(number_format) https://en.wikipedia.org/wiki/Unum_(number_format) http://www.johngustafson.net/pdfs/BeatingFloatingPoint.pdf http://www.johngustafson.net/pdfs/BeatingFloatingPoint.pdf (They are definitely not any easier to understand than the floating point system, though.)
- CodeArtisan 8y ago
  there is also DEC64 http://dec64.com/ http://dec64.com/
tasty_freeze 8y ago
When I went to university in 1982, one of the lower level courses was called "Numerical Methods". It went over all of the issues related to precision, stability, as well as a host of common numerical integration and approximation methods. I'm just a sample size of one, but isn't this kind of class a requirement for CS majors?
- Aeolun 8y ago
  Not in many of the CS related majors. Though I guess it is for hard CS.
  - jimmaswell 8y ago
    My CS course at a community college went over the limitations of floats in detail, in the CS class.
    - copperx 8y ago
      How detailed, if I may ask?
      - jimmaswell 8y ago
        I remember we learned how the representation worked bit-for-bit, and how it being stored in binary meant it couldn't perfectly represent everything in decimal. 1.01b meaning 12^0 + 02^-1 + 1*2^-2 for example.
- kccqzy 8y ago
  In my case, the CS class only covered the representation of floating point numbers (a sign bit, exponent bits, fraction bits, issues like bias etc) but not things like numerical approximation or integration methods. Those were in a separate class under the math department. And I think that's fair; after all those are really about scientific computing, not so much about computer science.
- dzhiurgis 8y ago
acqq 8y ago
> Even without moving away from the IEEE-754 standard, there are ways languages could be designed to minimize surprises. Floating point in base 10 is already in the standard since 2008: https://en.m.wikipedia.org/wiki/Decimal_floating_point https://en.m.wikipedia.org/wiki/Decimal_floating_point The standard is not to blame, the lack of demand for that feature is. Most of the potential users don’t know that they can demand from their software and hardware suppliers that feature. Using it there would be less “surprises.”
mark-r 8y ago
Showing that the result of 9999999999999999.0 – 9999999999999998.0 is a number between 1.9999999999999998 and 2.0000000000000124 will not solve the problem. IEEE floating point doesn't keep track of loss of precision.
- svat 8y ago
  To be clear, my point was that if programmers always saw floating-point numbers printed out as a range, from their beginning programming days, more of them would be likely to understand floating-point numbers better — or at least avoid the (impossible) idea that they map 1:1 with the real numbers. Having understood floating-point numbers, they would know what to expect from 9999999999999999.0 – 9999999999999998.0 with 64-bit floating-point. So though seeing a range here won't help magically restore precision that's been lost, having seen ranges earlier would help, before trying to carrying out this calculation. Nevertheless, you have a good point and what I take away from this is that showing a range for the end result of a computation (instead for a given number directly entered by the user/programmer) can be misleading if the result of the exact computation wouldn't actually have been in that range.

by svat 8y ago

crankylinuxuser 8y ago

When NASA can't even get it right, because of "surprises", there's no chance in hell I'm blaming us mere mortal programmers... or even 10x wizards. (0) It's time to look at other ways to depict fractional parts of numbers in a computer. I know that one can express any rational number as a integer fraction. And our computers are incapable of expressing a irrational number exactly - it does so to a certain precision... In other words, every number a computer expresses is a rational number. The exception is if the computer could express irrational numbers as symbolics, then we could work with the symbolic instead. And then as a last pass, the symbolic could convert to a imprecise rational depiction, or express as its native type. (0) https://itsfoss.com/a-floating-point-error-that-caused-a-damage-worth-half-a-billion/ https://itsfoss.com/a-floating-point-error-that-caused-a-dam...

saagarjha 8y ago
Rationals get unwieldy quickly, even with the simplest of arithmetic. A couple of additions is enough to get a large denominator.
- bjourne 8y ago
  That has not been my experience using a language with builtin support for rationals. The rational is simplified after each operation so it never grows unwieldy large. It is slower than floats, but imo vastly superior for most use cases.
  - Retra 8y ago
    "Most" use cases? You mean the use cases that don't involve heavy math, like graphics, physics, and statistics?
  - johndough 8y ago
    > The rational is simplified after each operation so it never grows unwieldy large. We recently had an exponential memory growth bug because rationals can not always be simplified, for example if you start with (2/3) and repeatedly square it. Fortunately, this was not in user-facing code, so there was no chance of a denial-of-service attack, but that's definitely something to watch out for with rationals.
  - stephencanon 8y ago
    Even simplifying after every operation, in the typical case exact rationals grow exponentially in the number of terms in the computation. This means that either: (a) you cannot use them for any non-trivial computation. (b) you have to round them, in which case they are strictly worse than floating-point numbers because they have redundant representations and a very non-uniform distribution.
empath75 8y ago
You can work directly in reals. http://hackage.haskell.org/package/exact-real http://hackage.haskell.org/package/exact-real

deleted 8y ago

[deleted]

sampo 8y ago

There is a proposal about "unum" or "posit" number system. They give more precision for small numbers (small meaning smaller than about 10^70, for 64bit numbers), less precision for huge numbers, and an overall larger range, than the floating point system. https://en.wikipedia.org/wiki/Unum_(number_format) https://en.wikipedia.org/wiki/Unum_(number_format) http://www.johngustafson.net/pdfs/BeatingFloatingPoint.pdf http://www.johngustafson.net/pdfs/BeatingFloatingPoint.pdf (They are definitely not any easier to understand than the floating point system, though.)

CodeArtisan 8y ago
there is also DEC64 http://dec64.com/ http://dec64.com/

tasty_freeze 8y ago

When I went to university in 1982, one of the lower level courses was called "Numerical Methods". It went over all of the issues related to precision, stability, as well as a host of common numerical integration and approximation methods. I'm just a sample size of one, but isn't this kind of class a requirement for CS majors?

Aeolun 8y ago
Not in many of the CS related majors. Though I guess it is for hard CS.
- jimmaswell 8y ago
  My CS course at a community college went over the limitations of floats in detail, in the CS class.
  - copperx 8y ago
    How detailed, if I may ask?
    - jimmaswell 8y ago
      I remember we learned how the representation worked bit-for-bit, and how it being stored in binary meant it couldn't perfectly represent everything in decimal. 1.01b meaning 12^0 + 02^-1 + 1*2^-2 for example.
kccqzy 8y ago
In my case, the CS class only covered the representation of floating point numbers (a sign bit, exponent bits, fraction bits, issues like bias etc) but not things like numerical approximation or integration methods. Those were in a separate class under the math department. And I think that's fair; after all those are really about scientific computing, not so much about computer science.
dzhiurgis 8y ago

acqq 8y ago

> Even without moving away from the IEEE-754 standard, there are ways languages could be designed to minimize surprises. Floating point in base 10 is already in the standard since 2008: https://en.m.wikipedia.org/wiki/Decimal_floating_point https://en.m.wikipedia.org/wiki/Decimal_floating_point The standard is not to blame, the lack of demand for that feature is. Most of the potential users don’t know that they can demand from their software and hardware suppliers that feature. Using it there would be less “surprises.”

mark-r 8y ago

Showing that the result of 9999999999999999.0 – 9999999999999998.0 is a number between 1.9999999999999998 and 2.0000000000000124 will not solve the problem. IEEE floating point doesn't keep track of loss of precision.

svat 8y ago
To be clear, my point was that if programmers always saw floating-point numbers printed out as a range, from their beginning programming days, more of them would be likely to understand floating-point numbers better — or at least avoid the (impossible) idea that they map 1:1 with the real numbers. Having understood floating-point numbers, they would know what to expect from 9999999999999999.0 – 9999999999999998.0 with 64-bit floating-point. So though seeing a range here won't help magically restore precision that's been lost, having seen ranges earlier would help, before trying to carrying out this calculation. Nevertheless, you have a good point and what I take away from this is that showing a range for the end result of a computation (instead for a given number directly entered by the user/programmer) can be misleading if the result of the exact computation wouldn't actually have been in that range.