5 ms·

Floor and Ceil versus Denormals on CPU and GPU

47 points by ibobev 5mo ago

crote 5mo ago
Another thing to keep in mind is that CPU processing of denormals tends to be extremely slow - I vaguely recall running into something like a 10x slowdown a decade ago. For a lot of applications the difference between a denormal and zero is small enough to be irrelevant, so if you expect near-zero values to be common, enabling a denormals-to-zero compiler flag might give you a pretty nice performance boost for free.
- adgjlsfhk1 5mo ago
  cpus that aren't Intel are plenty fast on denormals. Intel is the only one where denormals are 100x slower. (and Intel has fixed that on their new cpus, but only on their e cores)
- andrepd 5mo ago
  More like 100x, but not sure how true that is nowadays.
- mananaysiempre 5mo ago
  > CPU processing of denormals tends to be extremely slow - I vaguely recall running into something like a 10x slowdown a decade ago Intel CPU processing, where slowdowns can be as bad as couple hundred cycles. AMD CPUs penalize them much more mildly, usually single-digit cycles. (No idea about ARM.)
- adrian_b 5mo ago
  Denormal processing is slow only on certain CPUs, where the designers have been lazy, so when denormals are encountered that is handled by a microprogrammed sequence. During the last half of century there have been plenty of CPUs where denormals have been handled in hardware, so that any slow down caused by them is negligible. Except for generating graphic images seen by humans or in ML/AI applications, neither flushing results to zero nor treating denormal inputs as zero are acceptable, because they can lead to huge errors. Whoever fears that denormals can slow down an application, must enable the underflow exception. In that case denormals are never generated, but the underflow exceptions must be handled, because when denormals are not desired but underflows happen, that means that there are bugs in the program, which must be fixed. Denormals have been created so that people can mask the underflow exception and avoid to handle it, without dire consequences. However this habit of no longer handling the floating-point exceptions, like before the IEEE 754 standard, has created younger developers who are no longer aware of how FP arithmetic must be handled to avoid errors, so now there are too many who believe that the use of "-ffast-math" is permitted in general-purpose programs, not only in special applications where result accuracy does not matter. For correct results, you must use either denormals or underflow exception handling. There is no third choice. The third choice, like in GPUs, is only for when correctness is irrelevant.
kevmo314 5mo ago
> This is not the first time we can see Nvidia taking shortcuts to achieve maximum performance of their GPUs Why is implementing it correctly not performant? For context I have no idea how rounding is typically implemented anyways.
- adrian_b 5mo ago
  It is not correct because it does not implement the FP arithmetic standard and this can lead to much greater numerical errors than expected. NVIDIA is not responsible alone, because the Microsoft DirectX specification includes the non-standard behavior. Nevertheless, as shown in TFA, both the AMD and Intel GPUs allow the user to choose between correct behavior and incorrect behavior that might be faster, while NVIDIA ignores what the user requests and implements only the non-standard behavior. The developers of graphics or ML/AI applications do not care about errors, but there are also people who want to use GPUs for normal computations, where the accuracy of the results matters, so they want to be able to choose between correct behavior and incorrect but faster behavior. Actually "faster" is a misnomer, because denormals can be handled correctly without diminishing the speed, but that costs additional die area. Thus what NVIDIA gains by not implementing the right behavior is a reduced production cost.
  - SwuduSusuwu 5mo ago
    [dead]
yosefk 5mo ago
Flush denormals to zero. Even their inventor had trouble writing correct code in their presence - see the Appendix to that "what every programmer should know..." paper
- loicd 5mo ago
  > Even their inventor had trouble writing correct code in their presence I didn't know that. Could you provide a more specific reference?
- mananaysiempre 5mo ago
  On the other hand, they (unexpectedly to the inventor, who intended them to be a debugging tool) underpin a few foundational results in correctly rounded computation, such as https://en.wikipedia.org/wiki/Sterbenz_lemma https://en.wikipedia.org/wiki/Sterbenz_lemma.
andrepd 5mo ago
It's one of several issues with the design of IEEE floats, unfortunately. I wish we could start thinking more seriously about a new design, to complement if not replace IEEE in the long term. Posits are an example https://github.com/andrepd/posit-rust https://github.com/andrepd/posit-rust
- freeopinion 5mo ago
  Thank you for this contribution. Your repo has a link to the standard[0], which might interest some people. It makes me unreasonably happy to know that this was funded out of Singapore. [0] https://posithub.org/docs/posit_standard-2.pdf https://posithub.org/docs/posit_standard-2.pdf
- adrian_b 5mo ago
  The IEEE floats do not have any serious design issue. They are much better than any other floating-point formats that have ever been used. There have been many examples of badly designed floating-point numbers, e.g. those used by the IBM mainframes and those used by the DEC minicomputers. In both cases, minimizing the costs for computer vendors was prioritized over the properties needed by the end users. In scientific and technical computations, a constant relative error over the entire range of the numbers is optimal. Only logarithms would be better than IEEE floats, but the addition and subtraction of numbers stored as logarithms is too slow, so the IEEE floating-point numbers are a compromise between the speed of multiplication and the speed of addition. Posits redistribute the relative error over the range of numbers, increasing the precision of numbers close to unity by reducing the precision of numbers far from unity. This is a property that may seem desirable for input data and output data, but it is extremely undesirable for the intermediate values that are generated in computations. Even for input data and output data, in engineering there are many characteristic values of parts used in manufacturing, such as electronic components, where constant relative errors are needed over a range greater than 10^9 or even greater than 10^12 (e.g. the values of standard resistors, with a given tolerance, may vary from milliohms to gigaohms and those of standard capacitors from femtofarads to farads; it would not be acceptable to represent such nominal values as posits, with a tolerance depending on the nominal value). There may exist some applications where this is useful, but all such applications are among those that need low precision numbers, of 32 bits or less. So 16-bit or even 32-bit posits might be useful in certain circumstances, but it is pretty certain that 64-bit or bigger posits will always be inferior to IEEE floating-point numbers. The problem is that while the IEEE floating-point numbers are either optimal or at least acceptable in almost all applications, the fact that posits might be better only in certain special applications makes unlikely that the development of dedicated fast hardware for them can be worthwhile. Even if some applications might indeed like a better precision for numbers close to unity, for those applications posits must contend not only with floating-point numbers but also with fixed-point numbers, which do not need any special hardware and they can be implemented in any standard CPU with a simple software library. Fixed-point numbers have even better precision than posits for numbers close to unity, but posits have a greater range, where their precision drops quickly towards the extremities of the range. So the application domain of posits is squeezed between those of floating-point numbers and those of fixed-point numbers, leaving a very small number of applications where posits are optimal. I am not aware of any application that would justify the additional cost for posit-processing hardware. I think that the only chance for posits would be if someone would show that some posit format is better for certain ML/AI/LLM computations than narrow FP formats like BF16, FP8, FP4. That will be the only use case that could find people willing to pay enough money for the design of posit-processing hardware, in some kind of NPU for training or for inference. Like I have said, for technical/scientific computing posits are much inferior to FP64, graphic applications are happy with FP32 and FP16 and they do not have any reason for a change, while the applications that need high precision around unity are happy with fixed-point numbers and they also do not have any incentive to change.
Dwedit 5mo ago
Denormals happen to be the way that Zero can even be represented at all?
- aleph_minus_one 5mo ago
  For zeros (+0 and -0), there exist special representations in the IEEE 754 standard. Denormalized numbers are a different concept.
  - Dwedit 5mo ago
    Zero is a denormal number where the mantissa is zero. If you disallow denormals and treat them all as zero, then actual zero follows as well.
petermcneeley 5mo ago
WebGPU (WGSL) handles this by having a specified accuracy for each operation. https://www.w3.org/TR/WGSL/#concrete-float-accuracy https://www.w3.org/TR/WGSL/#concrete-float-accuracy This is all fully tested in the CTS. https://gpuweb.github.io/cts/standalone/?q=webgpu:shader,* https://gpuweb.github.io/cts/standalone/?q=webgpu:shader,*
SwuduSusuwu 5mo ago
[dead]
codedokode 5mo ago
Maybe we don't need denormals in most cases. Denormals are extremely tiny numbers (on order of 2^-149). It would be nice if RISC-V which aims for simpliciy, got rid of them. If you need such small numbers, I don't know, just use doubles or write your own floats.
- adrian_b 5mo ago
  Denormals are not needed if you are willing to handle underflow exceptions. Before Intel 8087 and the IEEE 754 standard, any decent floating-point unit generated overflow exceptions and underflow exceptions, which had to be handled by the programmer, unless the default behavior of crashing the program was acceptable. Intel 8087 and the standard based on it have offered to the lazy programmers the option to not handle the exceptions, in which case overflow exceptions generate infinities and the underflow exceptions generate denormals. When the exceptions are not handled, it is supposed that the programmer will check the final results of a long computation, and if infinities and denormals are not desired, but they exist nonetheless in the results, the programmer will investigate the reason and then the bug will be fixed. So anyone is free to ensure that no denormals will ever appear in an application , by enabling the underflow exception. If it is desired that the program must not crash, then the program must be written carefully, so that underflows are impossible. There is no correct way of eliminating denormals, except throwing exceptions on underflows. The flush denormals to zero on output and interpret denormals as zero on input behaviors are not permissible in any program that must produce correct results. Anyone who uses -ffast-math or similar options for compiling a program that is not intended for graphics or ML/AI, where errors supposedly do not matter, makes an unforgivable mistake. Unfortunately, "-ffast-math" enables a very large number of compilation options. A part of them are safe and they can cause a great increase in performance, like using fused multiply-add instructions. Others not only are dangerous, like flushing denormals to zero, but they also provide negligible performance increase on many processors. Therefore, instead of aggregate options like "-ffast-math" one must enable only some of the component options, for maximum performance, without affecting result accuracy. For example, in gcc and clang one must use "-ffp-contract=fast", for enabling FMA instructions.
  - rwallace 5mo ago

47 points by ibobev 5mo ago

crote 5mo ago

Another thing to keep in mind is that CPU processing of denormals tends to be extremely slow - I vaguely recall running into something like a 10x slowdown a decade ago. For a lot of applications the difference between a denormal and zero is small enough to be irrelevant, so if you expect near-zero values to be common, enabling a denormals-to-zero compiler flag might give you a pretty nice performance boost for free.

adgjlsfhk1 5mo ago
cpus that aren't Intel are plenty fast on denormals. Intel is the only one where denormals are 100x slower. (and Intel has fixed that on their new cpus, but only on their e cores)
andrepd 5mo ago
More like 100x, but not sure how true that is nowadays.
mananaysiempre 5mo ago
> CPU processing of denormals tends to be extremely slow - I vaguely recall running into something like a 10x slowdown a decade ago Intel CPU processing, where slowdowns can be as bad as couple hundred cycles. AMD CPUs penalize them much more mildly, usually single-digit cycles. (No idea about ARM.)
adrian_b 5mo ago
Denormal processing is slow only on certain CPUs, where the designers have been lazy, so when denormals are encountered that is handled by a microprogrammed sequence. During the last half of century there have been plenty of CPUs where denormals have been handled in hardware, so that any slow down caused by them is negligible. Except for generating graphic images seen by humans or in ML/AI applications, neither flushing results to zero nor treating denormal inputs as zero are acceptable, because they can lead to huge errors. Whoever fears that denormals can slow down an application, must enable the underflow exception. In that case denormals are never generated, but the underflow exceptions must be handled, because when denormals are not desired but underflows happen, that means that there are bugs in the program, which must be fixed. Denormals have been created so that people can mask the underflow exception and avoid to handle it, without dire consequences. However this habit of no longer handling the floating-point exceptions, like before the IEEE 754 standard, has created younger developers who are no longer aware of how FP arithmetic must be handled to avoid errors, so now there are too many who believe that the use of "-ffast-math" is permitted in general-purpose programs, not only in special applications where result accuracy does not matter. For correct results, you must use either denormals or underflow exception handling. There is no third choice. The third choice, like in GPUs, is only for when correctness is irrelevant.

kevmo314 5mo ago

> This is not the first time we can see Nvidia taking shortcuts to achieve maximum performance of their GPUs Why is implementing it correctly not performant? For context I have no idea how rounding is typically implemented anyways.

adrian_b 5mo ago
It is not correct because it does not implement the FP arithmetic standard and this can lead to much greater numerical errors than expected. NVIDIA is not responsible alone, because the Microsoft DirectX specification includes the non-standard behavior. Nevertheless, as shown in TFA, both the AMD and Intel GPUs allow the user to choose between correct behavior and incorrect behavior that might be faster, while NVIDIA ignores what the user requests and implements only the non-standard behavior. The developers of graphics or ML/AI applications do not care about errors, but there are also people who want to use GPUs for normal computations, where the accuracy of the results matters, so they want to be able to choose between correct behavior and incorrect but faster behavior. Actually "faster" is a misnomer, because denormals can be handled correctly without diminishing the speed, but that costs additional die area. Thus what NVIDIA gains by not implementing the right behavior is a reduced production cost.
- SwuduSusuwu 5mo ago
  [dead]

yosefk 5mo ago

Flush denormals to zero. Even their inventor had trouble writing correct code in their presence - see the Appendix to that "what every programmer should know..." paper

loicd 5mo ago
> Even their inventor had trouble writing correct code in their presence I didn't know that. Could you provide a more specific reference?
mananaysiempre 5mo ago
On the other hand, they (unexpectedly to the inventor, who intended them to be a debugging tool) underpin a few foundational results in correctly rounded computation, such as https://en.wikipedia.org/wiki/Sterbenz_lemma https://en.wikipedia.org/wiki/Sterbenz_lemma.

andrepd 5mo ago

It's one of several issues with the design of IEEE floats, unfortunately. I wish we could start thinking more seriously about a new design, to complement if not replace IEEE in the long term. Posits are an example https://github.com/andrepd/posit-rust https://github.com/andrepd/posit-rust

freeopinion 5mo ago
Thank you for this contribution. Your repo has a link to the standard[0], which might interest some people. It makes me unreasonably happy to know that this was funded out of Singapore. [0] https://posithub.org/docs/posit_standard-2.pdf https://posithub.org/docs/posit_standard-2.pdf
adrian_b 5mo ago
The IEEE floats do not have any serious design issue. They are much better than any other floating-point formats that have ever been used. There have been many examples of badly designed floating-point numbers, e.g. those used by the IBM mainframes and those used by the DEC minicomputers. In both cases, minimizing the costs for computer vendors was prioritized over the properties needed by the end users. In scientific and technical computations, a constant relative error over the entire range of the numbers is optimal. Only logarithms would be better than IEEE floats, but the addition and subtraction of numbers stored as logarithms is too slow, so the IEEE floating-point numbers are a compromise between the speed of multiplication and the speed of addition. Posits redistribute the relative error over the range of numbers, increasing the precision of numbers close to unity by reducing the precision of numbers far from unity. This is a property that may seem desirable for input data and output data, but it is extremely undesirable for the intermediate values that are generated in computations. Even for input data and output data, in engineering there are many characteristic values of parts used in manufacturing, such as electronic components, where constant relative errors are needed over a range greater than 10^9 or even greater than 10^12 (e.g. the values of standard resistors, with a given tolerance, may vary from milliohms to gigaohms and those of standard capacitors from femtofarads to farads; it would not be acceptable to represent such nominal values as posits, with a tolerance depending on the nominal value). There may exist some applications where this is useful, but all such applications are among those that need low precision numbers, of 32 bits or less. So 16-bit or even 32-bit posits might be useful in certain circumstances, but it is pretty certain that 64-bit or bigger posits will always be inferior to IEEE floating-point numbers. The problem is that while the IEEE floating-point numbers are either optimal or at least acceptable in almost all applications, the fact that posits might be better only in certain special applications makes unlikely that the development of dedicated fast hardware for them can be worthwhile. Even if some applications might indeed like a better precision for numbers close to unity, for those applications posits must contend not only with floating-point numbers but also with fixed-point numbers, which do not need any special hardware and they can be implemented in any standard CPU with a simple software library. Fixed-point numbers have even better precision than posits for numbers close to unity, but posits have a greater range, where their precision drops quickly towards the extremities of the range. So the application domain of posits is squeezed between those of floating-point numbers and those of fixed-point numbers, leaving a very small number of applications where posits are optimal. I am not aware of any application that would justify the additional cost for posit-processing hardware. I think that the only chance for posits would be if someone would show that some posit format is better for certain ML/AI/LLM computations than narrow FP formats like BF16, FP8, FP4. That will be the only use case that could find people willing to pay enough money for the design of posit-processing hardware, in some kind of NPU for training or for inference. Like I have said, for technical/scientific computing posits are much inferior to FP64, graphic applications are happy with FP32 and FP16 and they do not have any reason for a change, while the applications that need high precision around unity are happy with fixed-point numbers and they also do not have any incentive to change.

Dwedit 5mo ago

Denormals happen to be the way that Zero can even be represented at all?

aleph_minus_one 5mo ago
For zeros (+0 and -0), there exist special representations in the IEEE 754 standard. Denormalized numbers are a different concept.
- Dwedit 5mo ago
  Zero is a denormal number where the mantissa is zero. If you disallow denormals and treat them all as zero, then actual zero follows as well.

petermcneeley 5mo ago

WebGPU (WGSL) handles this by having a specified accuracy for each operation. https://www.w3.org/TR/WGSL/#concrete-float-accuracy https://www.w3.org/TR/WGSL/#concrete-float-accuracy This is all fully tested in the CTS. https://gpuweb.github.io/cts/standalone/?q=webgpu:shader,* https://gpuweb.github.io/cts/standalone/?q=webgpu:shader,*

SwuduSusuwu 5mo ago

[dead]

codedokode 5mo ago

Maybe we don't need denormals in most cases. Denormals are extremely tiny numbers (on order of 2^-149). It would be nice if RISC-V which aims for simpliciy, got rid of them. If you need such small numbers, I don't know, just use doubles or write your own floats.

adrian_b 5mo ago
Denormals are not needed if you are willing to handle underflow exceptions. Before Intel 8087 and the IEEE 754 standard, any decent floating-point unit generated overflow exceptions and underflow exceptions, which had to be handled by the programmer, unless the default behavior of crashing the program was acceptable. Intel 8087 and the standard based on it have offered to the lazy programmers the option to not handle the exceptions, in which case overflow exceptions generate infinities and the underflow exceptions generate denormals. When the exceptions are not handled, it is supposed that the programmer will check the final results of a long computation, and if infinities and denormals are not desired, but they exist nonetheless in the results, the programmer will investigate the reason and then the bug will be fixed. So anyone is free to ensure that no denormals will ever appear in an application , by enabling the underflow exception. If it is desired that the program must not crash, then the program must be written carefully, so that underflows are impossible. There is no correct way of eliminating denormals, except throwing exceptions on underflows. The flush denormals to zero on output and interpret denormals as zero on input behaviors are not permissible in any program that must produce correct results. Anyone who uses -ffast-math or similar options for compiling a program that is not intended for graphics or ML/AI, where errors supposedly do not matter, makes an unforgivable mistake. Unfortunately, "-ffast-math" enables a very large number of compilation options. A part of them are safe and they can cause a great increase in performance, like using fused multiply-add instructions. Others not only are dangerous, like flushing denormals to zero, but they also provide negligible performance increase on many processors. Therefore, instead of aggregate options like "-ffast-math" one must enable only some of the component options, for maximum performance, without affecting result accuracy. For example, in gcc and clang one must use "-ffp-contract=fast", for enabling FMA instructions.
- rwallace 5mo ago