5 ms·

Does anyone have examples of calculations that need quadruple precision? My last work place seemed fine using double precision for simulating satellites in orbi

Does anyone have examples of calculations that need quadruple precision? My last work place seemed fine using double precision for simulating satellites in orbit, and I have the imprecision double precision makes black holes and quantum physics simulation work correctly, so I have no intuition for this.

gh02t 2y ago
I see it in statistics. It's not uncommon in a lot of statistical algorithms to need to take the product (or sum) of a lot of very small numbers and then divide by that product when computing likelihoods of a bunch of independent measurements. There are some algebra tricks you can use to get around loss of precision like working in the log domain or some tedious scaling tricks, but working in quad precision can help when those aren't enough. The article also gives an example from quantum physics. Some chaotic or highly numerically unstable problems just need quad precision to not blow up; primarily problems/algorithms (can be the problem itself or the algorithm to solve it that causes the issues) that tend to catastrophically amplify small errors. Matrix inversion is another example, to the point that most algorithms try to do almost anything to avoid explicitly inverting a matrix. But sometimes you just have to.
- slashdave 2y ago
  Matrix inversion precision can be fixed iteratively (at a CPU cost, of course). Nearly singular matrices should probably be factored in any case.
  - gh02t 2y ago
    Probably, that falls into "doing almost anything to avoid inverting a matrix explicitly", but direct inversion is the classic example here.
    - kolbe 2y ago
      For numerical PDE solvers, you never do the actual inversion, but the discretization of the system (denoted A) can have significant sensitivity to numerical precision. Look up Condition Number and Preconditioning. In fact, resolving the problems with numerical precision for some A takes as much time as actually solving the system, and it's worth it.
      - gh02t 2y ago
        Oh I'm very aware, I'm lumping those all into "tricks used to avoid/work around the numerical issues associated with matrix inversion" for simplicity, because explicit computation of a matrix inverse is one of the classic examples of a numerically unstable task. Hence why a large subfield of applied math can be summarized as "coming up with ways to avoid matrix inversion." PDE solvers like you mention are one of the main applications for that. Tricks like clever factorization (which a lot of factorization algorithms have their own severe numerical issues e.g. some of the ways to compute QR or SVD), preconditioners, sparse and/or iterative algorithms like GMRES, randomized algorithms (my favorite) etc are all workarounds you wouldn't need if there was a fast and stable way to exactly invert any arbitrary non-singular matrix. Well, you would have less need for, there are other benefits to those methods but improving numerical stability by avoiding inverses is one of the main ones.
- Dylan16807 2y ago
  To put some specifics on "a lot of very small numbers", it's around a million numbers with part-per-billion precision, or a billion numbers with part-per-million precision, where you approach the limits of naive summation in double precision.
  - gh02t 2y ago
    For sums, yes, but products and division by small numbers are the real killer and those come up a lot in stats. Hence why you try to avoid them by working with log-probabilities (where products become sums), but sometimes you can't. Quad precision is a bit of a bandaid that just pushes the issue off when you don't have a better algorithm, but it works sometimes.
    - belkala 2y ago
      How so? All primitive operations are backward stable, and unlike addition and subtraction, division and multiplication are well-conditioned.
      - gh02t 2y ago
        It's more about resolution. Products of small numbers get small a lot faster than sums. Likewise for dividing two small numbers. The smallest possible quad precision number is a lot smaller than the smallest possible double precision one.
        Dylan16807 2y ago
        But the main feature of floating point is that you keep the same relative precision at all sizes. As long as you're not hitting infinity or denormals, multiplying doesn't lose information like adding a big number to a small number does. Do stats often deal with distict probabilities below 10^-300? And do they need to store them all over, or just in a couple variables that could do something clever?
        2y ago
gnulinux 2y ago
I use it for generating music through physical models. If the model requires a very numeric-sensitive algorithm like a differential equation solver, it's nice to have the extra precision, and usually performance doesn't matter too much in these sort of things (because ultimately, I just click "render" and wait anyway). To be very honest, I don't even know if I need 128bits, maybe a 32bit float would do... I never ran a comparative analysis. It just gives me more room for error, so that I can focus on the music/sound and not too much on the numerical analysis/programming. Working on hobby music+programming projects is the ultimate freedom.
- Dylan16807 2y ago
  I'm glad you're having fun but if you never compared it to double and single then you're not giving a valid example. Especially if you would have picked even higher if it was available.
- sfpotter 2y ago
  Basically guaranteed that you don't need quad for this.
- a1369209993 2y ago
  Of course, you also need to use solid gold audio cables, or you're just throwing away the extra precision via poor signal fidelity.
gmiller123456 2y ago
One situation that already uses two doubles, which you may already be familiar with, is the International Astronomical Union's Standards of Fundamental Astronomy library. It uses two doubles for the date in order to be able to be able to have pico/nano second precision over thousands of years. Of course the algorithms aren't precise to that level over thousands of years, but you still need the ability to specify it to that level during the time periods they are that accurate for.
- mitthrowaway2 2y ago
  Interesting. I would have thought that fixed-point would make more sense for a time quantity. If you use a 64-bit integer to represent picoseconds, it gives you a time range of about 18 million years.
  - Dylan16807 2y ago
    You calculated something wrong. 2^64 picoseconds is only half a year. Once you decide you need to go over 64 bits, the exact data type is largely a matter of convenience, and I can easily see two doubles being more convenient than either a quad or a 128-bit integer. You have more than 100 bits of precision in such a scheme, and it runs pretty fast.
    - mitthrowaway2 2y ago
      Ah good catch, my bad! 18 million seconds, not 18 million years. Oops =)
physicsguy 2y ago
You can easily get catastrophic cancellation in physical simulations, people are just not very aware of it. If for e.g. you see bare summation rather than some sort of sorting by magnitude then expect bad results as numbers get bigger.
- bee_rider 2y ago
  It is a bit weird though, because people also get very, very clever about designing their physical simulations so that they don’t have to care too much about their working precision.
  - physicsguy 2y ago
    Yeah, GPUs actually helped with this a lot when they first came out because suddenly to get the great performance you had to drop to single precision, at least until 2018 or so (can’t remember exactly now, but it used to be the case that the older cards were 2x as slow at DP)
    - bee_rider 2y ago
      I don’t really do GPU stuff, but I expect double to be at least 1/2 as slow as single, even on good hardware. I mean it is twice as many bits. (Maybe you think in bits/second, though, which is totally fair). I looked up my consumer card on a whim (playing with sprinkling some GPGPU into my code—cupy is actually really easy to use and ergonomic!), and apparently doubles are 1/32 as fast as singles. Yikes!
colechristensen 2y ago
Any long running calculation where the new state depends on the old state and there isn't much of a correction factor. Anything with "sensitive dependance on initial conditions", any dynamical systems. For example: simulating the three body problem (three celestial bodies of equalish size orbiting eachother). Very small differences get amplified, and any difference at all in how the math is implemented on different systems will result in divergent solutions.
mitthrowaway2 2y ago
It's a very quick and easy / lazy way to measure the floating-point error in your existing double-precision algorithms; run the computation at quadruple precision and check for differences in the output. If there's any significant difference then you take a closer look.

gh02t 2y ago

I see it in statistics. It's not uncommon in a lot of statistical algorithms to need to take the product (or sum) of a lot of very small numbers and then divide by that product when computing likelihoods of a bunch of independent measurements. There are some algebra tricks you can use to get around loss of precision like working in the log domain or some tedious scaling tricks, but working in quad precision can help when those aren't enough. The article also gives an example from quantum physics. Some chaotic or highly numerically unstable problems just need quad precision to not blow up; primarily problems/algorithms (can be the problem itself or the algorithm to solve it that causes the issues) that tend to catastrophically amplify small errors. Matrix inversion is another example, to the point that most algorithms try to do almost anything to avoid explicitly inverting a matrix. But sometimes you just have to.

slashdave 2y ago
Matrix inversion precision can be fixed iteratively (at a CPU cost, of course). Nearly singular matrices should probably be factored in any case.
- gh02t 2y ago
  Probably, that falls into "doing almost anything to avoid inverting a matrix explicitly", but direct inversion is the classic example here.
  - kolbe 2y ago
    For numerical PDE solvers, you never do the actual inversion, but the discretization of the system (denoted A) can have significant sensitivity to numerical precision. Look up Condition Number and Preconditioning. In fact, resolving the problems with numerical precision for some A takes as much time as actually solving the system, and it's worth it.
    - gh02t 2y ago
      Oh I'm very aware, I'm lumping those all into "tricks used to avoid/work around the numerical issues associated with matrix inversion" for simplicity, because explicit computation of a matrix inverse is one of the classic examples of a numerically unstable task. Hence why a large subfield of applied math can be summarized as "coming up with ways to avoid matrix inversion." PDE solvers like you mention are one of the main applications for that. Tricks like clever factorization (which a lot of factorization algorithms have their own severe numerical issues e.g. some of the ways to compute QR or SVD), preconditioners, sparse and/or iterative algorithms like GMRES, randomized algorithms (my favorite) etc are all workarounds you wouldn't need if there was a fast and stable way to exactly invert any arbitrary non-singular matrix. Well, you would have less need for, there are other benefits to those methods but improving numerical stability by avoiding inverses is one of the main ones.
Dylan16807 2y ago
To put some specifics on "a lot of very small numbers", it's around a million numbers with part-per-billion precision, or a billion numbers with part-per-million precision, where you approach the limits of naive summation in double precision.
- gh02t 2y ago
  For sums, yes, but products and division by small numbers are the real killer and those come up a lot in stats. Hence why you try to avoid them by working with log-probabilities (where products become sums), but sometimes you can't. Quad precision is a bit of a bandaid that just pushes the issue off when you don't have a better algorithm, but it works sometimes.
  - belkala 2y ago
    How so? All primitive operations are backward stable, and unlike addition and subtraction, division and multiplication are well-conditioned.
    - gh02t 2y ago
      It's more about resolution. Products of small numbers get small a lot faster than sums. Likewise for dividing two small numbers. The smallest possible quad precision number is a lot smaller than the smallest possible double precision one.
      - Dylan16807 2y ago
        But the main feature of floating point is that you keep the same relative precision at all sizes. As long as you're not hitting infinity or denormals, multiplying doesn't lose information like adding a big number to a small number does. Do stats often deal with distict probabilities below 10^-300? And do they need to store them all over, or just in a couple variables that could do something clever?
        2y ago

gnulinux 2y ago

I use it for generating music through physical models. If the model requires a very numeric-sensitive algorithm like a differential equation solver, it's nice to have the extra precision, and usually performance doesn't matter too much in these sort of things (because ultimately, I just click "render" and wait anyway). To be very honest, I don't even know if I need 128bits, maybe a 32bit float would do... I never ran a comparative analysis. It just gives me more room for error, so that I can focus on the music/sound and not too much on the numerical analysis/programming. Working on hobby music+programming projects is the ultimate freedom.

Dylan16807 2y ago
I'm glad you're having fun but if you never compared it to double and single then you're not giving a valid example. Especially if you would have picked even higher if it was available.
sfpotter 2y ago
Basically guaranteed that you don't need quad for this.
a1369209993 2y ago
Of course, you also need to use solid gold audio cables, or you're just throwing away the extra precision via poor signal fidelity.

gmiller123456 2y ago

One situation that already uses two doubles, which you may already be familiar with, is the International Astronomical Union's Standards of Fundamental Astronomy library. It uses two doubles for the date in order to be able to be able to have pico/nano second precision over thousands of years. Of course the algorithms aren't precise to that level over thousands of years, but you still need the ability to specify it to that level during the time periods they are that accurate for.

mitthrowaway2 2y ago
Interesting. I would have thought that fixed-point would make more sense for a time quantity. If you use a 64-bit integer to represent picoseconds, it gives you a time range of about 18 million years.
- Dylan16807 2y ago
  You calculated something wrong. 2^64 picoseconds is only half a year. Once you decide you need to go over 64 bits, the exact data type is largely a matter of convenience, and I can easily see two doubles being more convenient than either a quad or a 128-bit integer. You have more than 100 bits of precision in such a scheme, and it runs pretty fast.
  - mitthrowaway2 2y ago
    Ah good catch, my bad! 18 million seconds, not 18 million years. Oops =)

physicsguy 2y ago

You can easily get catastrophic cancellation in physical simulations, people are just not very aware of it. If for e.g. you see bare summation rather than some sort of sorting by magnitude then expect bad results as numbers get bigger.

bee_rider 2y ago
It is a bit weird though, because people also get very, very clever about designing their physical simulations so that they don’t have to care too much about their working precision.
- physicsguy 2y ago
  Yeah, GPUs actually helped with this a lot when they first came out because suddenly to get the great performance you had to drop to single precision, at least until 2018 or so (can’t remember exactly now, but it used to be the case that the older cards were 2x as slow at DP)
  - bee_rider 2y ago
    I don’t really do GPU stuff, but I expect double to be at least 1/2 as slow as single, even on good hardware. I mean it is twice as many bits. (Maybe you think in bits/second, though, which is totally fair). I looked up my consumer card on a whim (playing with sprinkling some GPGPU into my code—cupy is actually really easy to use and ergonomic!), and apparently doubles are 1/32 as fast as singles. Yikes!

colechristensen 2y ago

Any long running calculation where the new state depends on the old state and there isn't much of a correction factor. Anything with "sensitive dependance on initial conditions", any dynamical systems. For example: simulating the three body problem (three celestial bodies of equalish size orbiting eachother). Very small differences get amplified, and any difference at all in how the math is implemented on different systems will result in divergent solutions.

mitthrowaway2 2y ago

It's a very quick and easy / lazy way to measure the floating-point error in your existing double-precision algorithms; run the computation at quadruple precision and check for differences in the output. If there's any significant difference then you take a closer look.