4 ms·

The rise of mixed precision arithmetic

42 points by johndcook 11y ago

atemerev 11y ago
Still, nearly no progress in fast decimals, which are extremely important in financial applications. I'd even say that the only place where floating point is necessary is in simulations (physics, 3D, analog signals — all of it should be properly done with GPUs.) Everything else (2D layouts, finance, data processing) is better served with either rationals or decimals. We should remove floating point support from general-purpose CPUs and leave it to GPUs, where it belongs.
- chongli 11y ago
  We should remove floating point support from general-purpose CPUs and leave it to GPUs, where it belongs. The vast majority of CPU die real estate is devoted to mitigating the issues of constrained memory bandwidth and high memory latency. Taking out the FPUs is not going to make much difference. Not to mention the fact that you'd lose the ability to do floating point math in latency-sensitive environments, likely causing serious performance problems for whole classes of applications.
  - atemerev 11y ago
    On latency, you have a point. Too bad that Cell architecture didn't make it. Stream processing and reactive programming are all in vogue now.
- kazinator 11y ago
  Decimals are extremely important in financial applications. Fast decimals may range from unimportant to extremely important in specific situations, according to what is considered fast enough in those situations. > We should remove floating point support from general-purpose CPUs and leave it to GPUs, where it belongs. That's just silly. Let's move an extremely common and deeply integrated programming language feature onto special hardware. You do realize that "computer science" once actually consisted of little more than numerical analysis? Now you want to kick the latter off the computer. :)
  - SixSigma 11y ago
    My first PC was a 386 with a 387 floating point co-processor. [1] Intel chips have only had floating point support as default since 1989 with the 486DX [2] and still had a cheaper non-floating point version in the 486SX [3] Motorola 68000 was similarly constrained [1] https://en.wikipedia.org/wiki/X87 https://en.wikipedia.org/wiki/X87 [2] https://en.wikipedia.org/wiki/486DX https://en.wikipedia.org/wiki/486DX [3] https://en.wikipedia.org/wiki/486SX https://en.wikipedia.org/wiki/486SX
    - dalke 11y ago
      Which would suggest that on-chip floating point integration is important enough, across a wide range enough range of applications, to justify the expense.
      - atemerev 11y ago
        That was before GPUs. GPU is a modern co-processor.
        dalke 11y ago
        Is this just a terminology difference? The suggestion was to remove floating point from "general-purpose CPUs and leave it to GPUs". If GPUs are integrated into the CPU in the same way that a math-coprocessor is now part of the CPU, then the future "CPU" will be what we now think of as the "CPU + GPU" combination.
        dragonwriter 11y ago
        > That was before GPUs. GPU is a modern co-processor. And the trend of on-die GPUs as part of the CPU -- the same thing that happened with FPUs before them -- has already begun. Again showing that on-chip floating point integration (even of the kind of floating-point operations supported by GPUs as distinct from the functions already inherent in CPUs that have integrated FPUs) is important enough, across a wide range enough range of applications, to justify the expense.
        11y ago
- simonbyrne 11y ago
  There has been plenty of progress, you just have to pay a lot of money for it: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Power+Systems/page/POWER6+Decimal+Floating+Point+(DFP) https://www.ibm.com/developerworks/community/wikis/home?lang...
  - atemerev 11y ago
    Money is an option. Vendor lock-in isn't.
  - atemerev 11y ago
    I mean, come on! Proprietary C compiler on AIX?! Hardly sounds like "progress" to me.
- dahart 11y ago
  At first glance, this seems like such an extreme opinion that my curiosity is piqued. What do you want to see with fast decimals, what does that mean to you, in terms of hardware implementations? Do you think financial applications are more important in the hardware and software markets than simulations? ("Simulations", btw, includes all video games.) I'd like to give your suggestion the benefit of the doubt, that the "only" place floating point is necessary is in simulations and that GPUs could handle all floating point needs, but my gut reaction is that this is wildly out of touch with how often programs use floats. I might be wrong here, that's just my first thought. IIRC, GPUs across different devices often don't guarantee the level of precision you might need. Not to mention, getting data in and out, and using non built-in math functions, are both a pain. Depending on what kind of display or device or medium you're using, 2D layouts using floating point routinely. Most mobile devices now come in non 1:1 device pixel ratio. You have to use floating point to get pixel accurate layouts. Almost all vector art (SVG, PDF, etc) and almost all animation (CSS, Movie editing) use floating point heavily. "Data processing" - I dunno what that means, but a lot of the "data processing" I've done used floats or doubles or higher. Anything with computer vision in it will often use floating point; think about the rise of fingerprint and facial recognition. A long time ago, desktop CPUs used to come without floating point support, and everyone wanted it and paid for it, so now we have it. Granted that was before GPUs, but I'm curious if Intel would even entertain the idea of a general purpose desktop CPU without floating point support.
- Kristine1975 11y ago
  What about the non-GPU part of video games? What about audio processing?
  - atemerev 11y ago
    Audio processing is signal processing. Can be done on GPUs; the only reason it isn't is that CPUs are fast enough to not bother.
dzdt 11y ago
Is most of what you do numerical calculations? How much complexity are you willing to spend (in terms of development and maintanance programmer time) to buy a factor of 2 speedup to your code execution time in numerically heavy routines? Have you already optimised the hell out of those routines? Unless the answers are "yes", "a lot", and "yes", mixed precision arithmetic is not the answer for you.
natosaichek 11y ago
Rather than just adding more bits - doubles and double-doubles, I like the idea of moving around the boundary between the exponent and mantissa as necessary for the application, as well as adding the ability to determine how uncertain the number actually is. http://sites.ieee.org/scv-cs/files/2013/03/Right-SizingPrecision1.pdf http://sites.ieee.org/scv-cs/files/2013/03/Right-SizingPreci... www.amazon.com/The-End-Error-Computing-Computational/dp/1482239868
vorg 11y ago
> Arbitrary precision floating point arithmetic is available through [...] the core data type BigFloat in the new language Julia Go 1.5 added a big.Float type.

42 points by johndcook 11y ago

Still, nearly no progress in fast decimals, which are extremely important in financial applications. I'd even say that the only place where floating point is necessary is in simulations (physics, 3D, analog signals — all of it should be properly done with GPUs.) Everything else (2D layouts, finance, data processing) is better served with either rationals or decimals. We should remove floating point support from general-purpose CPUs and leave it to GPUs, where it belongs.

chongli 11y ago
We should remove floating point support from general-purpose CPUs and leave it to GPUs, where it belongs. The vast majority of CPU die real estate is devoted to mitigating the issues of constrained memory bandwidth and high memory latency. Taking out the FPUs is not going to make much difference. Not to mention the fact that you'd lose the ability to do floating point math in latency-sensitive environments, likely causing serious performance problems for whole classes of applications.
- atemerev 11y ago
  On latency, you have a point. Too bad that Cell architecture didn't make it. Stream processing and reactive programming are all in vogue now.
kazinator 11y ago
Decimals are extremely important in financial applications. Fast decimals may range from unimportant to extremely important in specific situations, according to what is considered fast enough in those situations. > We should remove floating point support from general-purpose CPUs and leave it to GPUs, where it belongs. That's just silly. Let's move an extremely common and deeply integrated programming language feature onto special hardware. You do realize that "computer science" once actually consisted of little more than numerical analysis? Now you want to kick the latter off the computer. :)
- SixSigma 11y ago
  My first PC was a 386 with a 387 floating point co-processor. [1] Intel chips have only had floating point support as default since 1989 with the 486DX [2] and still had a cheaper non-floating point version in the 486SX [3] Motorola 68000 was similarly constrained [1] https://en.wikipedia.org/wiki/X87 https://en.wikipedia.org/wiki/X87 [2] https://en.wikipedia.org/wiki/486DX https://en.wikipedia.org/wiki/486DX [3] https://en.wikipedia.org/wiki/486SX https://en.wikipedia.org/wiki/486SX
  - dalke 11y ago
    Which would suggest that on-chip floating point integration is important enough, across a wide range enough range of applications, to justify the expense.
    - atemerev 11y ago
      That was before GPUs. GPU is a modern co-processor.
      - dalke 11y ago
        Is this just a terminology difference? The suggestion was to remove floating point from "general-purpose CPUs and leave it to GPUs". If GPUs are integrated into the CPU in the same way that a math-coprocessor is now part of the CPU, then the future "CPU" will be what we now think of as the "CPU + GPU" combination.
      - dragonwriter 11y ago
        > That was before GPUs. GPU is a modern co-processor. And the trend of on-die GPUs as part of the CPU -- the same thing that happened with FPUs before them -- has already begun. Again showing that on-chip floating point integration (even of the kind of floating-point operations supported by GPUs as distinct from the functions already inherent in CPUs that have integrated FPUs) is important enough, across a wide range enough range of applications, to justify the expense.
      - 11y ago
simonbyrne 11y ago
There has been plenty of progress, you just have to pay a lot of money for it: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Power+Systems/page/POWER6+Decimal+Floating+Point+(DFP) https://www.ibm.com/developerworks/community/wikis/home?lang...
- atemerev 11y ago
  Money is an option. Vendor lock-in isn't.
- atemerev 11y ago
  I mean, come on! Proprietary C compiler on AIX?! Hardly sounds like "progress" to me.
dahart 11y ago
At first glance, this seems like such an extreme opinion that my curiosity is piqued. What do you want to see with fast decimals, what does that mean to you, in terms of hardware implementations? Do you think financial applications are more important in the hardware and software markets than simulations? ("Simulations", btw, includes all video games.) I'd like to give your suggestion the benefit of the doubt, that the "only" place floating point is necessary is in simulations and that GPUs could handle all floating point needs, but my gut reaction is that this is wildly out of touch with how often programs use floats. I might be wrong here, that's just my first thought. IIRC, GPUs across different devices often don't guarantee the level of precision you might need. Not to mention, getting data in and out, and using non built-in math functions, are both a pain. Depending on what kind of display or device or medium you're using, 2D layouts using floating point routinely. Most mobile devices now come in non 1:1 device pixel ratio. You have to use floating point to get pixel accurate layouts. Almost all vector art (SVG, PDF, etc) and almost all animation (CSS, Movie editing) use floating point heavily. "Data processing" - I dunno what that means, but a lot of the "data processing" I've done used floats or doubles or higher. Anything with computer vision in it will often use floating point; think about the rise of fingerprint and facial recognition. A long time ago, desktop CPUs used to come without floating point support, and everyone wanted it and paid for it, so now we have it. Granted that was before GPUs, but I'm curious if Intel would even entertain the idea of a general purpose desktop CPU without floating point support.
Kristine1975 11y ago
What about the non-GPU part of video games? What about audio processing?
- atemerev 11y ago
  Audio processing is signal processing. Can be done on GPUs; the only reason it isn't is that CPUs are fast enough to not bother.

dzdt 11y ago

Is most of what you do numerical calculations? How much complexity are you willing to spend (in terms of development and maintanance programmer time) to buy a factor of 2 speedup to your code execution time in numerically heavy routines? Have you already optimised the hell out of those routines? Unless the answers are "yes", "a lot", and "yes", mixed precision arithmetic is not the answer for you.

natosaichek 11y ago

Rather than just adding more bits - doubles and double-doubles, I like the idea of moving around the boundary between the exponent and mantissa as necessary for the application, as well as adding the ability to determine how uncertain the number actually is. http://sites.ieee.org/scv-cs/files/2013/03/Right-SizingPrecision1.pdf http://sites.ieee.org/scv-cs/files/2013/03/Right-SizingPreci... www.amazon.com/The-End-Error-Computing-Computational/dp/1482239868

vorg 11y ago

> Arbitrary precision floating point arithmetic is available through [...] the core data type BigFloat in the new language Julia Go 1.5 added a big.Float type.