4 ms·

re: "As long as they are small enough, floating point numbers can represent integers exactly." I recently discovered a wonderful floating point bug in the Appl

re: "As long as they are small enough, floating point numbers can represent integers exactly." I recently discovered a wonderful floating point bug in the Apple A7 GPU (fragment shader). I would pass a float from the vertex shader to the fragment shader, that was either 0.0, 1.0 or 2.0 and use that as an index to mask color channels. As such, I would convert to int and use it as an index. Problem was, the GPU would sometimes, on some random region of pixels, decide I didn't pass in 1.0, but 0.99999999999. Perform an int truncation, I have 0.0, I get the wrong index, my app starts flickering, I want to rip my hair out... Even on desktop machines, I had similar problems with large sets of tests that would produce different results depending on which (modern desktop) processor they ran on. Lesson: floating point is certainly dangerous for consistent representations that "matter". Regardless of a compiler decision or what the spec says.

deleted 12y ago
[deleted]
- baldajan 12y ago
  [Originally a reply to a statement that "1.0" should be reliable on systems because of the spec] Yeah, but all systems are made by humans, and all humans make mistakes - hence the GPU fragment shader error on the A7 that would occur sporadically over a "block-ish" region of pixels, even when I was drawing a square, values varied inside the square. Generally, these types of faults (in the fragment shader) are unnoticeable, since you can only output color, and not debug on a per value basis (at least in XCode). 0.9999999999 and 1.0 would produce the same color on the screen. So even if a fault exists, there would have little incentive to fix it.
sillysaurus3 12y ago
It's hard to understand how such an error could occur. Due to the nature of the floating point format, it seems very difficult for a bug to transform 1.0 into 0.999999. Floating point 1.0 has a sign bit of 0, an exponent of 0 (after bias), and a significand of 1.0. It's hard to imagine a sequence of transformations that could somehow result in getting 0.999999999 after conversion. I made a little "floating point binary explorer": http://ideone.com/61L4V6 http://ideone.com/61L4V6 (Apologies for the awful code quality; it's just a quick hack.) It prints each bit of a 32-bit float, which for 1.0 is "00111111 10000000 00000000 00000000". The first bit is the sign bit, the next 8 bits are the exponent, and the remaining 23 bits are the significand. So, sign=0, exponent=127, significand=0. The significand is actually 24-bit, not 23, and the leading "hidden bit" is always 1 (nitpick: unless the floating point value is 0.0 or -0.0). That means the significand in this case isn't actually zero, but "1 followed by 23 zeroes," so it's 2^24. The final value for a 32-bit float is: (significand / 2^24) * 2^(exponent - 127) That comes out to (2^24 / 2^24) * 2^(127 - 127), which is 1.0 * 2^0 which equals 1.0. For representing 2.0, everything is exactly the same, except the encoded exponent is 128 instead of 127. That comes out to (2^24 / 2^24) * 2^(128 - 127), or 1.0 * 2^1 which equals 2.0. Shader programs often use half precision floats (16-bit total, with 1 bit for sign, 5 bits for exponent, and 10 for significand) but the concepts should be exactly the same. 1.0 and 2.0 should always be 1.0 * 2^0 and 1.0 * 2^1 respectively, regardless of whether we're using half/single/double precision. I'm having trouble figuring out how your program wound up with 0.999999, even if there was a bug in the conversion. 32-bit floating point 0.999999 is "00111111 01111111 11111111 11101111" which is very different from the binary 1.0 float value: http://ideone.com/Qlqu4W http://ideone.com/Qlqu4W It has an exponent of 126 and almost the highest possible significand, which comes out to 1.9999 * 2^(126 - 127) = 1.9999 * 2^-1 = 0.9999. So in order for 1.0 to be erroneously transformed into 0.9999, the bug would need to invert all the bits of the significand, but subtract 1 from the exponent. Is there anything in particular that I should watch out for on the A7 GPU? I was just hoping to hear about any details you may have left out for simplicity, or any other factors that could've triggered the bug. > Even on desktop machines, I had similar problems with large sets of tests that would produce different results depending on which (modern desktop) processor they ran on. > Lesson: floating point is certainly dangerous for consistent representations that "matter". Regardless of a compiler decision or what the spec says. Are there floating point bugs on desktop processors? The only one I've heard of was the infamous Pentium FDIV bug http://en.wikipedia.org/wiki/Pentium_FDIV_bug http://en.wikipedia.org/wiki/Pentium_FDIV_bug and the only reason that wasn't hotfixed to match the spec was because the logic was hardwired into the chip rather than implemented in microcode, making a patch impossible. It seems like any deviation from the floating point standard in a modern desktop CPU would be noticed almost immediately, because there is a lot of code which relies on the exact IEEE floating point standard in order to function properly. There was recently a hash algorithm featured on HN which used floating point, and the only reason such an algorithm can be trusted is if every desktop processor properly adheres to the spec.
- acqq 12y ago
  I also suspect the 0.9999.. was an actual number in the commenter's inputs to the fragment shader. Unless he checked the binary representation of the numbers in question (in the input) he could have missed what's actually going on. Formatting and printing the number in base 10 hides the only real thing, which is the actual bits that represent the number.
  - jacquesm 12y ago
    It's not rare for GPUs to be pushed so close to the boundaries of what the underlying hardware can do that you'll see little bits of noise like this. For 'consumer' stuff this usually does not matter, but I think it is one of the reasons (besides revenues) that Nvidia has a 'compute' line and a 'display' line. Even if they're based on the exact same tech it would not surprise me one bit if the safety margins were eroded considerably on the consumer stuff and if it wasn't tested as good before shipping (on the chip level).
    - sillysaurus3 12y ago
      ATI would probably be a better example. Nvidia has historically been more precise and has implemented specs pretty faithfully. ATI has of course become AMD, and the reputation of modern AMD graphics cards among developers seems to be pretty great. It would surprise me if sending the value 1.0 from a vertex program to fragment program ever resulted in anything except 1.0 on a desktop GPU. It's not unusual to truncate a float in a fragment program and the use the resulting integer as an index. I think Nvidia has a fairly rigorous battery of tests which they use to ensure compliance with specs. The tests check to make sure that values are being interpolated correctly, to check for off-by-one errors at the edges of triangles, and other such corner cases. The tests are automated, because once you implement a graphics spec correctly, you can generate an output image which should be the same every time it's generated. That way, if an error creeps into the driver, it should be caught pretty quickly, since it will cause a deviation in the rendered output examined by the test suite. I'm not sure how often a problem occurs in an Nvidia consumer card but not their 'compute' line. Nvidia tries to use the same driver codebase across as many of their products as possible. Here's an interesting thread where someone points out that certain OpenCL code was giving the correct result on x86 and Nvidia cards, but not AMD cards: http://devgurus.amd.com/thread/145582 http://devgurus.amd.com/thread/145582 In that thread, it's suggested that the reason Nvidia gave the correct result was because their CUDA compute specs had better error margins compared to OpenCL specs. So in that case, it would appear that Nvidia's compute line actually enhanced their accuracy in other contexts such as OpenCL. Rather than implement a different driver for OpenCL, CUDA, and consumer graphics, it seems that Nvidia uses the same driver to power all three, which seems to enhance accuracy rather than hinder it. While writing up this comment and researching this, I stumbled across this interesting paper from 2011 that details exactly what programmers can expect when programming to an Nvidia GPU (in this case using CUDA): https://developer.nvidia.com/sites/default/files/akamai/cuda/files/NVIDIA-CUDA-Floating-Point.pdf https://developer.nvidia.com/sites/default/files/akamai/cuda... ... It's a pretty fun read which helps to solidify the impression that Nvidia cares deeply about providing accurate floating point results. I also came across this article which was too awesome not to mention: http://randomascii.wordpress.com/2013/07/16/floating-point-determinism/ http://randomascii.wordpress.com/2013/07/16/floating-point-d... ... It's not really related to GPU floating point, but it's an excellent exploration of floating point determinism and the effect of various floating point settings. For mobile graphics stacks like those in cellphones / iPads etc, I share your skepticism. I'm not sure how rigorously tested mobile GPUs are, or how faithfully those stacks adhere to specs. I was hoping to get more info from the original commenter about how to reproduce the bug in the A7 to get a better idea of what to watch out for on mobile.
deleted 12y ago
[deleted]
userbinator 12y ago
Even on desktop machines, I had similar problems with large sets of tests that would produce different results depending on which (modern desktop) processor they ran on. If you're talking about x86 and can isolate a test case, then I'm sure Intel and AMD would be very, very interested in hearing about it. Even better if it is reproducible in something common like Windows' calculator. Despite the fact that people think of FP math as "inexact" the truth is that it is theoretically exact in the sense that all the arithmetic operations are defined in terms of simple operations on bits just like with integer math (although with more complex combinations of operations), and there is no nondeterminism - the same operation on the same operands should always yield the same bit representations for all combinations of operands. The only inexactness comes from the use of binary fractions, but it is still deterministic. I suppose for GPUs there is a line of reasoning that says "if the viewer won't notice the diference, then maybe it doesn't matter and we can accept a quantifiably nonzero error rate" but this would be absolutely unacceptable for GPGPU, and I very strongly hope designers of general-purpose FPUs (like x86 and ARM) never go down this dangerous path. (Of course, you should rule out things like overclocked hardware causing these differences.)

deleted 12y ago

[deleted]

baldajan 12y ago
[Originally a reply to a statement that "1.0" should be reliable on systems because of the spec] Yeah, but all systems are made by humans, and all humans make mistakes - hence the GPU fragment shader error on the A7 that would occur sporadically over a "block-ish" region of pixels, even when I was drawing a square, values varied inside the square. Generally, these types of faults (in the fragment shader) are unnoticeable, since you can only output color, and not debug on a per value basis (at least in XCode). 0.9999999999 and 1.0 would produce the same color on the screen. So even if a fault exists, there would have little incentive to fix it.

sillysaurus3 12y ago

It's hard to understand how such an error could occur. Due to the nature of the floating point format, it seems very difficult for a bug to transform 1.0 into 0.999999. Floating point 1.0 has a sign bit of 0, an exponent of 0 (after bias), and a significand of 1.0. It's hard to imagine a sequence of transformations that could somehow result in getting 0.999999999 after conversion. I made a little "floating point binary explorer": http://ideone.com/61L4V6 http://ideone.com/61L4V6 (Apologies for the awful code quality; it's just a quick hack.) It prints each bit of a 32-bit float, which for 1.0 is "00111111 10000000 00000000 00000000". The first bit is the sign bit, the next 8 bits are the exponent, and the remaining 23 bits are the significand. So, sign=0, exponent=127, significand=0. The significand is actually 24-bit, not 23, and the leading "hidden bit" is always 1 (nitpick: unless the floating point value is 0.0 or -0.0). That means the significand in this case isn't actually zero, but "1 followed by 23 zeroes," so it's 2^24. The final value for a 32-bit float is: (significand / 2^24) * 2^(exponent - 127) That comes out to (2^24 / 2^24) * 2^(127 - 127), which is 1.0 * 2^0 which equals 1.0. For representing 2.0, everything is exactly the same, except the encoded exponent is 128 instead of 127. That comes out to (2^24 / 2^24) * 2^(128 - 127), or 1.0 * 2^1 which equals 2.0. Shader programs often use half precision floats (16-bit total, with 1 bit for sign, 5 bits for exponent, and 10 for significand) but the concepts should be exactly the same. 1.0 and 2.0 should always be 1.0 * 2^0 and 1.0 * 2^1 respectively, regardless of whether we're using half/single/double precision. I'm having trouble figuring out how your program wound up with 0.999999, even if there was a bug in the conversion. 32-bit floating point 0.999999 is "00111111 01111111 11111111 11101111" which is very different from the binary 1.0 float value: http://ideone.com/Qlqu4W http://ideone.com/Qlqu4W It has an exponent of 126 and almost the highest possible significand, which comes out to 1.9999 * 2^(126 - 127) = 1.9999 * 2^-1 = 0.9999. So in order for 1.0 to be erroneously transformed into 0.9999, the bug would need to invert all the bits of the significand, but subtract 1 from the exponent. Is there anything in particular that I should watch out for on the A7 GPU? I was just hoping to hear about any details you may have left out for simplicity, or any other factors that could've triggered the bug. > Even on desktop machines, I had similar problems with large sets of tests that would produce different results depending on which (modern desktop) processor they ran on. > Lesson: floating point is certainly dangerous for consistent representations that "matter". Regardless of a compiler decision or what the spec says. Are there floating point bugs on desktop processors? The only one I've heard of was the infamous Pentium FDIV bug http://en.wikipedia.org/wiki/Pentium_FDIV_bug http://en.wikipedia.org/wiki/Pentium_FDIV_bug and the only reason that wasn't hotfixed to match the spec was because the logic was hardwired into the chip rather than implemented in microcode, making a patch impossible. It seems like any deviation from the floating point standard in a modern desktop CPU would be noticed almost immediately, because there is a lot of code which relies on the exact IEEE floating point standard in order to function properly. There was recently a hash algorithm featured on HN which used floating point, and the only reason such an algorithm can be trusted is if every desktop processor properly adheres to the spec.

acqq 12y ago
I also suspect the 0.9999.. was an actual number in the commenter's inputs to the fragment shader. Unless he checked the binary representation of the numbers in question (in the input) he could have missed what's actually going on. Formatting and printing the number in base 10 hides the only real thing, which is the actual bits that represent the number.
- jacquesm 12y ago
  It's not rare for GPUs to be pushed so close to the boundaries of what the underlying hardware can do that you'll see little bits of noise like this. For 'consumer' stuff this usually does not matter, but I think it is one of the reasons (besides revenues) that Nvidia has a 'compute' line and a 'display' line. Even if they're based on the exact same tech it would not surprise me one bit if the safety margins were eroded considerably on the consumer stuff and if it wasn't tested as good before shipping (on the chip level).
  - sillysaurus3 12y ago
    ATI would probably be a better example. Nvidia has historically been more precise and has implemented specs pretty faithfully. ATI has of course become AMD, and the reputation of modern AMD graphics cards among developers seems to be pretty great. It would surprise me if sending the value 1.0 from a vertex program to fragment program ever resulted in anything except 1.0 on a desktop GPU. It's not unusual to truncate a float in a fragment program and the use the resulting integer as an index. I think Nvidia has a fairly rigorous battery of tests which they use to ensure compliance with specs. The tests check to make sure that values are being interpolated correctly, to check for off-by-one errors at the edges of triangles, and other such corner cases. The tests are automated, because once you implement a graphics spec correctly, you can generate an output image which should be the same every time it's generated. That way, if an error creeps into the driver, it should be caught pretty quickly, since it will cause a deviation in the rendered output examined by the test suite. I'm not sure how often a problem occurs in an Nvidia consumer card but not their 'compute' line. Nvidia tries to use the same driver codebase across as many of their products as possible. Here's an interesting thread where someone points out that certain OpenCL code was giving the correct result on x86 and Nvidia cards, but not AMD cards: http://devgurus.amd.com/thread/145582 http://devgurus.amd.com/thread/145582 In that thread, it's suggested that the reason Nvidia gave the correct result was because their CUDA compute specs had better error margins compared to OpenCL specs. So in that case, it would appear that Nvidia's compute line actually enhanced their accuracy in other contexts such as OpenCL. Rather than implement a different driver for OpenCL, CUDA, and consumer graphics, it seems that Nvidia uses the same driver to power all three, which seems to enhance accuracy rather than hinder it. While writing up this comment and researching this, I stumbled across this interesting paper from 2011 that details exactly what programmers can expect when programming to an Nvidia GPU (in this case using CUDA): https://developer.nvidia.com/sites/default/files/akamai/cuda/files/NVIDIA-CUDA-Floating-Point.pdf https://developer.nvidia.com/sites/default/files/akamai/cuda... ... It's a pretty fun read which helps to solidify the impression that Nvidia cares deeply about providing accurate floating point results. I also came across this article which was too awesome not to mention: http://randomascii.wordpress.com/2013/07/16/floating-point-determinism/ http://randomascii.wordpress.com/2013/07/16/floating-point-d... ... It's not really related to GPU floating point, but it's an excellent exploration of floating point determinism and the effect of various floating point settings. For mobile graphics stacks like those in cellphones / iPads etc, I share your skepticism. I'm not sure how rigorously tested mobile GPUs are, or how faithfully those stacks adhere to specs. I was hoping to get more info from the original commenter about how to reproduce the bug in the A7 to get a better idea of what to watch out for on mobile.

deleted 12y ago

[deleted]

userbinator 12y ago

Even on desktop machines, I had similar problems with large sets of tests that would produce different results depending on which (modern desktop) processor they ran on. If you're talking about x86 and can isolate a test case, then I'm sure Intel and AMD would be very, very interested in hearing about it. Even better if it is reproducible in something common like Windows' calculator. Despite the fact that people think of FP math as "inexact" the truth is that it is theoretically exact in the sense that all the arithmetic operations are defined in terms of simple operations on bits just like with integer math (although with more complex combinations of operations), and there is no nondeterminism - the same operation on the same operands should always yield the same bit representations for all combinations of operands. The only inexactness comes from the use of binary fractions, but it is still deterministic. I suppose for GPUs there is a line of reasoning that says "if the viewer won't notice the diference, then maybe it doesn't matter and we can accept a quantifiably nonzero error rate" but this would be absolutely unacceptable for GPGPU, and I very strongly hope designers of general-purpose FPUs (like x86 and ARM) never go down this dangerous path. (Of course, you should rule out things like overclocked hardware causing these differences.)