4 ms·

In this context, I also highly recommend John Gustafson's book, The End of Error. It is full of great examples, and at one point contrasts advances in numerical

by triska 8y ago

In this context, I also highly recommend John Gustafson's book, The End of Error. It is full of great examples, and at one point contrasts advances in numerical computing with improvements in other areas: "In 1970, a printer might produce something that looks like this, and take about 30 seconds to do so: [picture] Over forty years later, a laser printer still might take 30 seconds to put out a single page, but technology has improved to allow full-color, high-resolution output." The book makes a compelling argument that working with floating point numbers is similar to printing thousands of low-quality pages per second, instead of getting a single high-quality page in 30 seconds. A few paragraphs later, the book states: "Eventually it becomes so obvious that change is necessary that a wrenching shift finally takes place. Such a shift is overdue for numerical computing, where we are stuck with antediluvian tools. Once we make the shift, it will become possible to solve problems that have baffled us for decades. The evidence supporting that claim is here in this book, which contains quite a few results that appear to be new." For more information, see: https://www.crcpress.com/The-End-of-Error-Unum-Computing/Gustafson/p/book/9781482239867 https://www.crcpress.com/The-End-of-Error-Unum-Computing/Gus... and John Gustafson's home page: http://johngustafson.net/ http://johngustafson.net/

nestorD 8y ago
He has great examples. Sadly he found shortcomings to his unum numbers and his new alternative[0] do not seem that great compared to the current status quo. [0] https://posithub.org/index https://posithub.org/index
jedbrown 8y ago
>90% of numerical analysis would remain if continuous arithmetic was exact. There exist some prominent failures of finite precision arithmetic (and they resonate with uninformed audiences), but discretization and modeling errors are far more insidious and a deeper challenge for reliable scientific computing.
- angry_octet 8y ago
  Agreed. Though I'd say that model errors are a challenge, but I wouldn't call them insidious. There is generally serious analysis of the effect of the chosen discretization method/grid size, how to handle sub-grid scale effects, etc. Likewise models. I think it is interesting to compare this old panic with ML work. Without gazillions of flops the neural network approach was dead. After a few years it became clear that float/double precision was useless, and now for certain problems half precision is standard. Where one resource (people, CPU, ram, money) is the fixed constraint, other factors will be tweaked to compensate.
  - jedbrown 8y ago
    I don't know what domain you work in. The modest 1986 editorial statement from the Journal of Fluids Engineering is still pretty much a gold standard for rigor in real-world applications and is exceedingly rarely achieved in domains where direct observational data is hard to come by (most of geophysics, much of biophysics, etc.). E.g., https://jedbrown.org/files/20160912-NumericalAccuracy.pdf https://jedbrown.org/files/20160912-NumericalAccuracy.pdf
jcranmer 8y ago
unums are at best very controversial. In actuality, I don't think I've heard a numerical analyst (save for Gustafson) come out in favor of them. The original proposal that Gustafson had involved a variable-width format, which is going to absolutely kill performance in practice (particularly on the GPU-based supercomputers that are all the rage nowadays). The posit format was essentially a recognition that variable-width doesn't work. The more serious error is that, well, interval arithmetic turns out not to be useful in practice. Essentially, the potential accumulation of error is exponential in the number of steps in the worst case, so interval arithmetic in practice too often gives you the answer "the result is 1, but it could have been anywhere between -inf and +inf", which is completely useless. He may complain that he doesn't want to use a numerical analyst to get work done, but the rebuttal is that numerical analysis is still necessary anyways, so promising that one isn't necessary is a false promise.
- zlynx 8y ago
  At least with (-inf, +inf) you know that the answer is nonsense and that you need to go back to the algorithm to find a better order of operations. With ordinary floating point, all too often people don't realize their results are nonsense. Adding measurement and computation error bounds to their graphs is illuminating.
  - triska 8y ago
    This is indeed a key point in John Gustafson's arguments: With floats, even though there are specialized flags and values that let you detect certain problematic situations, these flags are often not readily accessible in programming languages, and hence these issues tend to go unnoticed.
    - jcranmer 8y ago
      > these flags are often not readily accessible in programming languages Unless you use C, C++, or Fortran. Which accounts for most numerical algorithms code. Yes, most people aren't aware of them, but not being aware of all the functions in the standard library isn't the same as them not being present there.

by triska 8y ago

nestorD 8y ago

He has great examples. Sadly he found shortcomings to his unum numbers and his new alternative[0] do not seem that great compared to the current status quo. [0] https://posithub.org/index https://posithub.org/index

jedbrown 8y ago

>90% of numerical analysis would remain if continuous arithmetic was exact. There exist some prominent failures of finite precision arithmetic (and they resonate with uninformed audiences), but discretization and modeling errors are far more insidious and a deeper challenge for reliable scientific computing.

angry_octet 8y ago
Agreed. Though I'd say that model errors are a challenge, but I wouldn't call them insidious. There is generally serious analysis of the effect of the chosen discretization method/grid size, how to handle sub-grid scale effects, etc. Likewise models. I think it is interesting to compare this old panic with ML work. Without gazillions of flops the neural network approach was dead. After a few years it became clear that float/double precision was useless, and now for certain problems half precision is standard. Where one resource (people, CPU, ram, money) is the fixed constraint, other factors will be tweaked to compensate.
- jedbrown 8y ago
  I don't know what domain you work in. The modest 1986 editorial statement from the Journal of Fluids Engineering is still pretty much a gold standard for rigor in real-world applications and is exceedingly rarely achieved in domains where direct observational data is hard to come by (most of geophysics, much of biophysics, etc.). E.g., https://jedbrown.org/files/20160912-NumericalAccuracy.pdf https://jedbrown.org/files/20160912-NumericalAccuracy.pdf

jcranmer 8y ago

unums are at best very controversial. In actuality, I don't think I've heard a numerical analyst (save for Gustafson) come out in favor of them. The original proposal that Gustafson had involved a variable-width format, which is going to absolutely kill performance in practice (particularly on the GPU-based supercomputers that are all the rage nowadays). The posit format was essentially a recognition that variable-width doesn't work. The more serious error is that, well, interval arithmetic turns out not to be useful in practice. Essentially, the potential accumulation of error is exponential in the number of steps in the worst case, so interval arithmetic in practice too often gives you the answer "the result is 1, but it could have been anywhere between -inf and +inf", which is completely useless. He may complain that he doesn't want to use a numerical analyst to get work done, but the rebuttal is that numerical analysis is still necessary anyways, so promising that one isn't necessary is a false promise.

zlynx 8y ago
At least with (-inf, +inf) you know that the answer is nonsense and that you need to go back to the algorithm to find a better order of operations. With ordinary floating point, all too often people don't realize their results are nonsense. Adding measurement and computation error bounds to their graphs is illuminating.
- triska 8y ago
  This is indeed a key point in John Gustafson's arguments: With floats, even though there are specialized flags and values that let you detect certain problematic situations, these flags are often not readily accessible in programming languages, and hence these issues tend to go unnoticed.
  - jcranmer 8y ago
    > these flags are often not readily accessible in programming languages Unless you use C, C++, or Fortran. Which accounts for most numerical algorithms code. Yes, most people aren't aware of them, but not being aware of all the functions in the standard library isn't the same as them not being present there.