4 ms·

Subnormals in particular I think are a pretty big failure of the IEEE standard, in the sense that because their implementation is very slow & varies across arch

Subnormals in particular I think are a pretty big failure of the IEEE standard, in the sense that because their implementation is very slow & varies across architectures, the only time most people learn about them is when they're hitting FP perf issues & want to disable their support.

adgjlsfhk1 4y ago
yeah. the ieee standard was way better than not having it, but I think that for a lot of cases, posits are much better thought out now. hopefully they'll start to gain broader hardware support.
- Aardwolf 4y ago
  What makes posits so good for hardware? Does the fact that they have variable sized exponent not cause similar difficulty as the subnormals in floats?
  - adgjlsfhk1 4y ago
    The main difference of posits (the variable sized exponent) is slightly worse for hardware (but not much). Posits, however win back a lot of points because they remove a lot of special cases. FP hardware generally needs special checks for NaNs, Infs, -0 and subnormals. Posits only have 2 "special" numbers 0 and NaR. The problem with subnormal floats isn't that they are inherently slow, it's that they are different and rare which means they aren't well optimized for. With Posits, everything is the same, which reduces the number of branches in hardware.
- adrian_b 4y ago
  Posits have advantages for low-precision numbers, e.g. up to 32 bits. For high-precision numbers, e.g. for 64 bits, the standard double-precision floating-point numbers are the best. Any number format using a given number of bits has the same number of values that can be represented. The only difference between various formats, e.g. integers, fixed-point numbers, floating-point numbers or posits, is how the same number of points is distributed over the line of the real numbers, being more dense in some parts and more rare in other parts. For scientific computation, the optimal distribution of the representable numbers is when the relative error is constant. The ideal number format would be logarithmic, but it is too difficult to add and subtract logarithmic numbers, so the second best format is used, i.e. floating-point numbers, which allow fast implementations for all important arithmetic operations. Posits have a non-uniform relative error, low close too 1 and high for large and small numbers, so they are unsuitable for complex physics modelling, which needs more computations with large numbers and small numbers than with numbers close to 1. On the other hand, for the applications where low-precision numbers are suitable, e.g. machine learning, graphics or some kinds of DSP, posits could be better than floating-point numbers.
  - adgjlsfhk1 4y ago
    I agree with you that Posits only really make sense for <=32 bits. That said, I disagree that the optimal distribution of numbers is logarithmic. Especially for 16 bit or smaller computations, if you are dealing with big or small numbers, you need to renormalize to roughly 1 anyway to prevent overflow and underflow. As such, it makes a lot more sense to have extra accuracy near 1.
    - adrian_b 4y ago
      I agree with you that for small precision logarithmic distribution may not be optimal and that is why posits may be a better choice for such applications. Logarithmic distribution is optimal for complex physics models where there are relationships between many different kinds of physical quantities, so it is not possible or convenient to choose a scale factor that would make all quantities have values close to 1. In such applications many numbers are very large or very small and any other distribution except logarithmic leads to increased errors in the results. Such models also typically need a 64-bit precision. Moreover, having a much greater exponent range than provided by the 32-bit floating-point format, in order to avoid overflows and underflows, is frequently even more important for them than having a greater precision. If for an application the range of the 32-bit floating-point numbers is good enough, then there are indeed chances that posits might be better for it.
adrian_b 4y ago
Their implementation is not necessarily slow. In the past there have been many implementations where using subnormals did not cause any speed penalty. The real disadvantage appears to be that a fast implementation is likely to be more expensive, i.e. it requires more complex circuits, thus a greater chip area. Most designers of recent CPUs or GPUs appear to be much more concerned with obtaining a greater speed and a minimum chip area than with obtaining adequate numeric accuracy, because most buyers are influenced by speed benchmark results and know little or nothing about the accuracy required by their computations. Moreover, the subnormals are an optional feature of the floating-point standard, the alternative to using subnormals is to enable the underflow exception. However, an overwhelming majority of the programmers are too lazy to enable and handle overflow exceptions, much less to enable and handle underflow exceptions. Unlike using either underflow exceptions or subnormals, the use of flush-to-zero and denormals-are-zero is acceptable only in the programs where computation errors do not matter at all, such as games.
- mkbosmans 4y ago
  > Unlike using either underflow exceptions or subnormals, the use of flush-to-zero and denormals-are-zero > is acceptable only in the programs where computation errors do not matter at all, such as games. That is not true. FTZ and DAZ are perfectly reasonable in a lot of scientific computing scenario's, where computation errors in general are closely scrutinized.
  - adrian_b 4y ago
    It depends on what is meant by "scrutinized". FTZ and DAZ and any other methods that are certain to introduce errors can be accepted only inside a computation for which there is an independent way to verify the results and that way is always used, without exceptions. For example, it is indeed OK to use FTZ and DAZ when solving a system of equations, if, and only if, after obtaining a solution it is inserted in the original equations and a residual error is computed, this time without using FTZ and DAZ, and an excessive residual error triggers an appropriate means of notifying the user about the failure.

adgjlsfhk1 4y ago

yeah. the ieee standard was way better than not having it, but I think that for a lot of cases, posits are much better thought out now. hopefully they'll start to gain broader hardware support.

Aardwolf 4y ago
What makes posits so good for hardware? Does the fact that they have variable sized exponent not cause similar difficulty as the subnormals in floats?
- adgjlsfhk1 4y ago
  The main difference of posits (the variable sized exponent) is slightly worse for hardware (but not much). Posits, however win back a lot of points because they remove a lot of special cases. FP hardware generally needs special checks for NaNs, Infs, -0 and subnormals. Posits only have 2 "special" numbers 0 and NaR. The problem with subnormal floats isn't that they are inherently slow, it's that they are different and rare which means they aren't well optimized for. With Posits, everything is the same, which reduces the number of branches in hardware.
adrian_b 4y ago
Posits have advantages for low-precision numbers, e.g. up to 32 bits. For high-precision numbers, e.g. for 64 bits, the standard double-precision floating-point numbers are the best. Any number format using a given number of bits has the same number of values that can be represented. The only difference between various formats, e.g. integers, fixed-point numbers, floating-point numbers or posits, is how the same number of points is distributed over the line of the real numbers, being more dense in some parts and more rare in other parts. For scientific computation, the optimal distribution of the representable numbers is when the relative error is constant. The ideal number format would be logarithmic, but it is too difficult to add and subtract logarithmic numbers, so the second best format is used, i.e. floating-point numbers, which allow fast implementations for all important arithmetic operations. Posits have a non-uniform relative error, low close too 1 and high for large and small numbers, so they are unsuitable for complex physics modelling, which needs more computations with large numbers and small numbers than with numbers close to 1. On the other hand, for the applications where low-precision numbers are suitable, e.g. machine learning, graphics or some kinds of DSP, posits could be better than floating-point numbers.
- adgjlsfhk1 4y ago
  I agree with you that Posits only really make sense for <=32 bits. That said, I disagree that the optimal distribution of numbers is logarithmic. Especially for 16 bit or smaller computations, if you are dealing with big or small numbers, you need to renormalize to roughly 1 anyway to prevent overflow and underflow. As such, it makes a lot more sense to have extra accuracy near 1.
  - adrian_b 4y ago
    I agree with you that for small precision logarithmic distribution may not be optimal and that is why posits may be a better choice for such applications. Logarithmic distribution is optimal for complex physics models where there are relationships between many different kinds of physical quantities, so it is not possible or convenient to choose a scale factor that would make all quantities have values close to 1. In such applications many numbers are very large or very small and any other distribution except logarithmic leads to increased errors in the results. Such models also typically need a 64-bit precision. Moreover, having a much greater exponent range than provided by the 32-bit floating-point format, in order to avoid overflows and underflows, is frequently even more important for them than having a greater precision. If for an application the range of the 32-bit floating-point numbers is good enough, then there are indeed chances that posits might be better for it.

adrian_b 4y ago

Their implementation is not necessarily slow. In the past there have been many implementations where using subnormals did not cause any speed penalty. The real disadvantage appears to be that a fast implementation is likely to be more expensive, i.e. it requires more complex circuits, thus a greater chip area. Most designers of recent CPUs or GPUs appear to be much more concerned with obtaining a greater speed and a minimum chip area than with obtaining adequate numeric accuracy, because most buyers are influenced by speed benchmark results and know little or nothing about the accuracy required by their computations. Moreover, the subnormals are an optional feature of the floating-point standard, the alternative to using subnormals is to enable the underflow exception. However, an overwhelming majority of the programmers are too lazy to enable and handle overflow exceptions, much less to enable and handle underflow exceptions. Unlike using either underflow exceptions or subnormals, the use of flush-to-zero and denormals-are-zero is acceptable only in the programs where computation errors do not matter at all, such as games.

mkbosmans 4y ago
> Unlike using either underflow exceptions or subnormals, the use of flush-to-zero and denormals-are-zero > is acceptable only in the programs where computation errors do not matter at all, such as games. That is not true. FTZ and DAZ are perfectly reasonable in a lot of scientific computing scenario's, where computation errors in general are closely scrutinized.
- adrian_b 4y ago
  It depends on what is meant by "scrutinized". FTZ and DAZ and any other methods that are certain to introduce errors can be accepted only inside a computation for which there is an independent way to verify the results and that way is always used, without exceptions. For example, it is indeed OK to use FTZ and DAZ when solving a system of equations, if, and only if, after obtaining a solution it is inserted in the original equations and a residual error is computed, this time without using FTZ and DAZ, and an excessive residual error triggers an appropriate means of notifying the user about the failure.