13 ms·

> Of all the float-pointing point numbers your computer can represent, a quarter of them lie in [0,1]. Many people think floating point numbers are magically p

> Of all the float-pointing point numbers your computer can represent, a quarter of them lie in [0,1]. Many people think floating point numbers are magically precise. They're not. They're far more accurate at lower magnitudes and precision fades as you work with larger values.

semaphoreP 10y ago
Or you can think of it as precision in the number of significant digits for a number. Each floating point number can only store a certain amount of significant digits. The fractional precision on each float is the same, but the absolute magnitude of the error in the float value increases as the value increases.
- mirimir 10y ago
  That's how I learned it.
jacobolus 10y ago
John Gustafson’s new “sigmoid unum”/“posit” proposal is kind of interesting; see his recent talk: http://web.stanford.edu/class/ee380/Abstracts/170201.html http://web.stanford.edu/class/ee380/Abstracts/170201.html https://news.ycombinator.com/item?id=13562164 https://news.ycombinator.com/item?id=13562164 It does a variable number of fraction bits, so that the values near 1 are even more densely represented than under the usual IEEE floats, while also providing greater dynamic range (but at reduced precision). I made a little diagram showing a visual comparison of every represented value in a toy 8-bit version vs. IEEE-style floats: https://groups.google.com/group/unum-computing/attach/e80274981e971/posit_vs_float.pdf?part=0.1&authuser=0 https://groups.google.com/group/unum-computing/attach/e80274...
- sunfish 10y ago
  Out of curiosity, I started watching this talk, but right there in the introduction, the very first example is a dot product where supposedly IEEE 754 double precision gets the wrong answer; I stopped to check the result, and I got the correct answer with double precision (even without binary sum collapse). Then, he says the x87's results are nondeterministic due to being affected by cache, which is incorrect. Then, he trots out a series of half-truths about IEEE 754, which are somewhat true but don't mean what he's presenting them to mean. He even cites a Cray 1 quirk as an example of IEEE 754 weirdness, when in reality, the Cray 1 (1975) predated IEEE 754 by about 10 years. Designing a good arithmetic system takes a lot of attention to detail, and he doesn't make a good impression by being that loose with details in his introduction.
  - dbcurtis 10y ago
    The Cray 1 was done before anyone realized the importance of denormals. And denormals are gruesome to implement on vector machines anyway.
    - greglindahl 10y ago
      Only a subset of scientific codes benefit from denormals, and it does not appear that implementing them is that big of a deal, given that 4-stage pipelines can do it. The pipeline depth to memory is a lot bigger than 4.
      - dbcurtis 10y ago
        If you consider every iterative algorithm that solves systems of nonlinear equations a subset you cam ignore.... SPICE, linpack, fishpack.... sorry, denormals are essential to today's algorithms. As to "no big deal".... show me your code. Don't worry, I'll be able to understand it. After 20 years as a CPU designer I've learned how to understand bit-bashing.
        greglindahl 10y ago
        The iterative algorithms for all of the scientific fields that I've worked on don't fall within the subset that you're describing, so no, I'm not going to agree that you're speaking about the majority of today's algorithms. Weather, climate, finite differencing, FFT... nope. Implicit systems, sure. Radar cross-sections, sure. And as a CPU designer, are you disagreeing that most FP units today take 4 cycles (fully pipelined) (add and multiply, of course)? And that main memory is a lot farther away than that? Given 20 years of experience, you missed out on the Cray PVP machines that were the start of this sub-thread. But the cycle counts I'm giving are modern ones. Cray did eventually implement IEEE in these machines without any significant problems, but that was more than 20 years ago.
      - nkurz 10y ago
        Only a subset of scientific codes benefit from denormals, and it does not appear that implementing them is that big of a deal, given that 4-stage pipelines can do it. Maybe I'm misunderstanding your terminology, but it seems like you are saying that operations involving denormals have the same latency as normal floating point multiplications and additions. At least for multiplication on Intel chips through Haswell, I think the current case is that subnormals between 0 and FLT_MIN still have abysmal performance --- 100+ cycles of penalty. Here's Bruce Dawson from a few years ago: Performance implications on SSE Intel handles NaNs and infinities much better on their SSE FPUs than on their x87 FPUs. NaNs and infinities have long been handled at full speed on this floating-point unit. However denormals are still a problem. On Core 2 processors the worst-case I have measured is a 175 times slowdown, on SSE addition and multiplication. On SandyBridge Intel has fixed this for addition – I was unable to produce any slowdown on ‘addps’ instructions. However SSE multiplication (‘mulps’) on Sandybridge has about a 140 cycle penalty if one of the inputs or results is a denormal. https://randomascii.wordpress.com/2012/05/20/thats-not-normalthe-performance-of-odd-floats/ https://randomascii.wordpress.com/2012/05/20/thats-not-norma... And here's the overview from a slightly more recent paper: C. Subnormal Performance Variability Due to the complex nature of the floating point numbers, processors struggle to handle certain inputs efficiently. In particular, it is well understood that operating on subnormal values can cause extreme performance issues, including slowdowns of up to 100× [19]. As an example, on a Core i7 processor using SSE instructions, performing standard mul- tiply between two normal numbers takes 4 clock cycles, whereas the same multiply given a subnormal input takes over 200 clock cycles. https://cseweb.ucsd.edu/~hovav/dist/subnormal.pdf https://cseweb.ucsd.edu/~hovav/dist/subnormal.pdf Are you saying this has been fixed in recent (or non-Intel) chips? Or maybe you were considering only NaN and Inf when you said denormals?
RandomOpinion 10y ago
>Many people think floating point numbers are magically precise. Even worse, it kicks in earlier than one might naively think. People are often surprised to hear a value as simple as 0.1 has no matching floating point representation. (For those interested, http://www.exploringbinary.com/why-0-point-1-does-not-exist-in-floating-point/ http://www.exploringbinary.com/why-0-point-1-does-not-exist-... has a nicely illustrated explanation.)
- TD-Linux 10y ago
  Being unable to represent 0.1 exactly is unrelated to any precision limitation. It's because the quantization of binary and decimal is different.
- greenleafjacob 10y ago
  This isn't true with e.g. decimal64.
  - astrodust 10y ago
    Those sorts of numbers aren't floating point, they're typically fixed point.
    - SAI_Peregrinus 10y ago
      No, that's explicitly a floating point format. It's one of the representations in IEE-754-2008. The three decimal floating point formats are decimal32, decimal64, and decimal128. That people often use them where fixed-point math would work doesn't mean they're not floating point formats.
ryandrake 10y ago
Correct me if I'm wrong, but my understanding is that a consequence of this is that it's better to store, for example, angles as radians from -pi to pi rather than as degrees between 0.0 and 360.0, in order to take advantage of all that precision between 0 and 1.
- jzwinck 10y ago
  I don't see why that would help. Of course there is more precision in terms of decimal places around 1 than around 360. But if you use 360 each 1 represents a much smaller piece of the circle. To put it another way...it's about significant digits. A float has about 7 and a double about 15. It does not matter whether you use 0 to 360 or 0 to 3.6 million...the number of digits that are meaningful remain the same.
  - mirimir 10y ago
    For binning data in spreadsheets for histograms etc, I typically use text() to set significant digits, and then value(). It's also good sometimes to make sure that 0 is really 0.
- janekm 10y ago
  Even 'better' just store them in fixed point representation.
  - astrodust 10y ago
    If all you're dealing with is angles, 64-bit quantization of the 360° space will give you far more accuracy than you'll ever need, and it will be absolutely uniform throughout.
    - janekm 10y ago
      True enough... my mind is in the embedded space where say a good enough representation in say 16-bit is desirable.
    - 10y ago
dasfasf 10y ago
Every number system in which all numbers occupy the same amount of space must have this property (namely, that there are roughly as many positive numbers above 1 as below 1, and therefore that there is greater precision near 0) or the accuracy of the function x->1/x will suffer.
moron4hire 10y ago
What's hilarious is that this is what is meant by calling them "floating point". The counterpoint "fixed point", in comparison, does not necessarily imply integers.

semaphoreP 10y ago

Or you can think of it as precision in the number of significant digits for a number. Each floating point number can only store a certain amount of significant digits. The fractional precision on each float is the same, but the absolute magnitude of the error in the float value increases as the value increases.

mirimir 10y ago
That's how I learned it.

jacobolus 10y ago

John Gustafson’s new “sigmoid unum”/“posit” proposal is kind of interesting; see his recent talk: http://web.stanford.edu/class/ee380/Abstracts/170201.html http://web.stanford.edu/class/ee380/Abstracts/170201.html https://news.ycombinator.com/item?id=13562164 https://news.ycombinator.com/item?id=13562164 It does a variable number of fraction bits, so that the values near 1 are even more densely represented than under the usual IEEE floats, while also providing greater dynamic range (but at reduced precision). I made a little diagram showing a visual comparison of every represented value in a toy 8-bit version vs. IEEE-style floats: https://groups.google.com/group/unum-computing/attach/e80274981e971/posit_vs_float.pdf?part=0.1&authuser=0 https://groups.google.com/group/unum-computing/attach/e80274...

sunfish 10y ago
Out of curiosity, I started watching this talk, but right there in the introduction, the very first example is a dot product where supposedly IEEE 754 double precision gets the wrong answer; I stopped to check the result, and I got the correct answer with double precision (even without binary sum collapse). Then, he says the x87's results are nondeterministic due to being affected by cache, which is incorrect. Then, he trots out a series of half-truths about IEEE 754, which are somewhat true but don't mean what he's presenting them to mean. He even cites a Cray 1 quirk as an example of IEEE 754 weirdness, when in reality, the Cray 1 (1975) predated IEEE 754 by about 10 years. Designing a good arithmetic system takes a lot of attention to detail, and he doesn't make a good impression by being that loose with details in his introduction.
- dbcurtis 10y ago
  The Cray 1 was done before anyone realized the importance of denormals. And denormals are gruesome to implement on vector machines anyway.
  - greglindahl 10y ago
    Only a subset of scientific codes benefit from denormals, and it does not appear that implementing them is that big of a deal, given that 4-stage pipelines can do it. The pipeline depth to memory is a lot bigger than 4.
    - dbcurtis 10y ago
      If you consider every iterative algorithm that solves systems of nonlinear equations a subset you cam ignore.... SPICE, linpack, fishpack.... sorry, denormals are essential to today's algorithms. As to "no big deal".... show me your code. Don't worry, I'll be able to understand it. After 20 years as a CPU designer I've learned how to understand bit-bashing.
      - greglindahl 10y ago
        The iterative algorithms for all of the scientific fields that I've worked on don't fall within the subset that you're describing, so no, I'm not going to agree that you're speaking about the majority of today's algorithms. Weather, climate, finite differencing, FFT... nope. Implicit systems, sure. Radar cross-sections, sure. And as a CPU designer, are you disagreeing that most FP units today take 4 cycles (fully pipelined) (add and multiply, of course)? And that main memory is a lot farther away than that? Given 20 years of experience, you missed out on the Cray PVP machines that were the start of this sub-thread. But the cycle counts I'm giving are modern ones. Cray did eventually implement IEEE in these machines without any significant problems, but that was more than 20 years ago.
    - nkurz 10y ago
      Only a subset of scientific codes benefit from denormals, and it does not appear that implementing them is that big of a deal, given that 4-stage pipelines can do it. Maybe I'm misunderstanding your terminology, but it seems like you are saying that operations involving denormals have the same latency as normal floating point multiplications and additions. At least for multiplication on Intel chips through Haswell, I think the current case is that subnormals between 0 and FLT_MIN still have abysmal performance --- 100+ cycles of penalty. Here's Bruce Dawson from a few years ago: Performance implications on SSE Intel handles NaNs and infinities much better on their SSE FPUs than on their x87 FPUs. NaNs and infinities have long been handled at full speed on this floating-point unit. However denormals are still a problem. On Core 2 processors the worst-case I have measured is a 175 times slowdown, on SSE addition and multiplication. On SandyBridge Intel has fixed this for addition – I was unable to produce any slowdown on ‘addps’ instructions. However SSE multiplication (‘mulps’) on Sandybridge has about a 140 cycle penalty if one of the inputs or results is a denormal. https://randomascii.wordpress.com/2012/05/20/thats-not-normalthe-performance-of-odd-floats/ https://randomascii.wordpress.com/2012/05/20/thats-not-norma... And here's the overview from a slightly more recent paper: C. Subnormal Performance Variability Due to the complex nature of the floating point numbers, processors struggle to handle certain inputs efficiently. In particular, it is well understood that operating on subnormal values can cause extreme performance issues, including slowdowns of up to 100× [19]. As an example, on a Core i7 processor using SSE instructions, performing standard mul- tiply between two normal numbers takes 4 clock cycles, whereas the same multiply given a subnormal input takes over 200 clock cycles. https://cseweb.ucsd.edu/~hovav/dist/subnormal.pdf https://cseweb.ucsd.edu/~hovav/dist/subnormal.pdf Are you saying this has been fixed in recent (or non-Intel) chips? Or maybe you were considering only NaN and Inf when you said denormals?

RandomOpinion 10y ago

>Many people think floating point numbers are magically precise. Even worse, it kicks in earlier than one might naively think. People are often surprised to hear a value as simple as 0.1 has no matching floating point representation. (For those interested, http://www.exploringbinary.com/why-0-point-1-does-not-exist-in-floating-point/ http://www.exploringbinary.com/why-0-point-1-does-not-exist-... has a nicely illustrated explanation.)

TD-Linux 10y ago
Being unable to represent 0.1 exactly is unrelated to any precision limitation. It's because the quantization of binary and decimal is different.
greenleafjacob 10y ago
This isn't true with e.g. decimal64.
- astrodust 10y ago
  Those sorts of numbers aren't floating point, they're typically fixed point.
  - SAI_Peregrinus 10y ago
    No, that's explicitly a floating point format. It's one of the representations in IEE-754-2008. The three decimal floating point formats are decimal32, decimal64, and decimal128. That people often use them where fixed-point math would work doesn't mean they're not floating point formats.

ryandrake 10y ago

Correct me if I'm wrong, but my understanding is that a consequence of this is that it's better to store, for example, angles as radians from -pi to pi rather than as degrees between 0.0 and 360.0, in order to take advantage of all that precision between 0 and 1.

jzwinck 10y ago
I don't see why that would help. Of course there is more precision in terms of decimal places around 1 than around 360. But if you use 360 each 1 represents a much smaller piece of the circle. To put it another way...it's about significant digits. A float has about 7 and a double about 15. It does not matter whether you use 0 to 360 or 0 to 3.6 million...the number of digits that are meaningful remain the same.
- mirimir 10y ago
  For binning data in spreadsheets for histograms etc, I typically use text() to set significant digits, and then value(). It's also good sometimes to make sure that 0 is really 0.
janekm 10y ago
Even 'better' just store them in fixed point representation.
- astrodust 10y ago
  If all you're dealing with is angles, 64-bit quantization of the 360° space will give you far more accuracy than you'll ever need, and it will be absolutely uniform throughout.
  - janekm 10y ago
    True enough... my mind is in the embedded space where say a good enough representation in say 16-bit is desirable.
  - 10y ago

dasfasf 10y ago

Every number system in which all numbers occupy the same amount of space must have this property (namely, that there are roughly as many positive numbers above 1 as below 1, and therefore that there is greater precision near 0) or the accuracy of the function x->1/x will suffer.

moron4hire 10y ago

What's hilarious is that this is what is meant by calling them "floating point". The counterpoint "fixed point", in comparison, does not necessarily imply integers.