17 ms·

Honestly, I think IEEE 754 floating point numbers are a pretty bad way of dealing with real numbers and just cause tons of headaches that every math library has

Honestly, I think IEEE 754 floating point numbers are a pretty bad way of dealing with real numbers and just cause tons of headaches that every math library has to deal with. Even high level programmers aren't shielded from the NaN nonsense. It would be great if we had an underlying implementation we could ignore and that allowed us to think at a more mathematical level. I actually found something recently on this topic while googling for any alternatives, its called Unum and looks very interesting: https://en.wikipedia.org/wiki/Unum_(number_format) https://en.wikipedia.org/wiki/Unum_(number_format) Does anyone know more about this? Are there any chips out there that support it?

com2kid 10y ago
Just reading the list of disadvantages, the format sounds interesting but not that practical in a world where massive numbers of FPUs operating within a thermal limit is a primary design concern.
Retra 10y ago
Wikipedia provides a pretty good summary: William Kahan claims the following issues with unums: 1. Unum computation does not always deliver correct results. 2. The description of unums sidesteps using unum for solving calculus problems. 3. Unums can be expensive in terms of time and power consumption. 4. Each computation in unum space is likely to change the bit length of the structure. This requires either unpacking them into a fixed-size space, or data allocation, deallocation, and garbage collection during unum operations, similar to the issues for dealing with variable-length records in mass storage. 5. Unums provide only two kinds of numerical exception, quiet and signaling NaN (Not-a-Number). 6. Unum computation may deliver overly loose bounds from the selection of an algebraically correct but numerically unstable algorithm. 7. The costs and benefits of unum over short precision floating point for problems requiring low precision are not obvious. 8. Solving differential equations and evaluating integrals with unums guarantee correct answers but may not be as fast as methods that usually work. Point #4 makes implementation tricky and slow. Point #6 (from what I understand) means unum operations tend to guarantee correctness by being overly vague - by denoting results that FP can produce using ranges that are just too big to be useful.
gajjanag 10y ago
> Does anyone know more about this? Are there any chips out there that support it? Not an answer to your question, but I found out recently that there is definitely significant interest in ideas like interval arithmetic in hardware among mathematicians. The application I am somewhat familiar with is "brute forcing" proofs of inequalities, which can get very annoying, even for apparently easy stuff like 2 variable ones. Indeed, "The Princeton Companion to Applied Mathematics" edited by N. Higham et al has an entire article on this (in section II.20). In particular, the article has a reference to a library called filib++ (http://www2.math.uni-wuppertal.de/~xsc/software/filib.html http://www2.math.uni-wuppertal.de/~xsc/software/filib.html). I unfortunately have no idea how it performs/how good it is. By the way, thanks a lot for the Unum reference; seems like it is still in too early a stage for inclusion in the book above.
- delhanty 10y ago
  Floating point is Nick Higham's thing isn't? I have his book "Accuracy and Stability of Numerical Algorithms, Second Edition" http://www.ma.man.ac.uk/~higham/asna/ http://www.ma.man.ac.uk/~higham/asna/ but I haven't got very far with it yet. Maybe the article in the Princeton Companion is a summarized version of his book?
- jacobolus 10y ago
  I emailed Nick Higham asking about Unums a year and a half ago. Hopefully he doesn’t mind me quoting from private email: > I have Gustafson's book and have read a lot of it. I don't yet know what to make of it. [..] His Unum system is a completely different way of working to floating point, so all I can say is that standard error analysis techniques will probably not be applicable. > The book does not yet seem to have atttracted much attention in the numerical analysis community.
nine_k 10y ago
If NaN is nonsense, what would you have 1/0 or acos(2) to produce? (An exception is the same NaN, only less convenientlying packaged.)
- rnhmjoj 10y ago
  Ideally I would restrict all functions to their natural domain using refinement types. This ways 1/0 simply doesn't type check and you don't have to bother assigning it some nonsensical value. However I can see something like this becoming much more a hassle than a benefit when things get complicated enough.
  - brianberns 10y ago
    I'm not very familiar with refinement types. How would that work? For example, how would you write a function that answers the mean value of an array of numbers? You could define Array.Length to be of type NonNegativeInteger, but that still includes 0. Alternatively, you'd have to define a NonEmptyArray type that excludes arrays of length 0, but that seems less than useful.
    - rnhmjoj 10y ago
      Yes, that's exactly how it works. See this article for an example: http://nikita-volkov.github.io/refined/ http://nikita-volkov.github.io/refined/ The thing is that, instead of redefining everything yourself, you could use a library that provides you with a set of predicates to compose and build the refinement you need. There are also much sophisticated ways to do this, like Liquid Haskell.
    - protomikron 10y ago
      > Alternatively, you'd have to define a NonEmptyArray type that excludes arrays of length 0, but that seems less than useful. I think the usefulness of a specific NonEmptyArray type is controversial, and I would say such types could be extremely useful. AFAIK there have been discussions if Haskell's Prelude (part of the standard library providing a name space for convenient functions and types, similar to the functions you get in e.g. Python like map, zip, filter, range, etc.) should be more save by not advocating too generic types and functions like the standard `map`, `head` and `tail` implementations. Functions like `head` and `tail` can be seen as problematic as they can fail and cause runtime errors (the head and tail of an empty list cause a runtime error), but I do not think they already changed that (or if it's even a topic of discussion at the moment). It's a little bit a matter of taste. Think about Unix' "head" program that by default prints maximal 10 lines of a file. How should we deal with the following? $ touch empty.txt $ head -n1 empty.txt Currently this works - you do not get output and $? == 0, but one could also say "head" should panic here.
jcranmer 10y ago
A cursory examination of the documentation for and against the unum stuff seems to indicate a vehement ad hominem argument running between Gustafson and Kahan, which makes it somewhat hard to find objective information. The unum stuff does seem to be greatly oversold, though. Fundamentally, there is a problem in mathematics: some equations naturally blow up tiny uncertainties into major ones. Indeed, in chaotic systems, such uncertainties can be extremely difficult to describe accurately. Floating point numbers would give you an implausibly precise but meaninglessly inaccurate number, whereas unums would indicate that the answer is basically everything. The difference here basically boils down to Kahan arguing that you should do numerical analysis first to figure out if the result will be meaningful, while Gustafson wants to throw calculus and numerical analysis out the window. Surprisingly, the dispute is less about the actual format than the style of numerical algorithms. Gustafson seems to be advocating an iterative, increasing-precision approach to solving problems like root finding in lieu of standard numerical techniques. Kahan's retort is that the state-of-the-art techniques are more than sufficient, and that Gustafson's techniques take longer to achieve acceptable results. I didn't see a response from Gustafson to the latter accusation--the slide deck I see has him shifting from an argument about the power usage involved in storing 64 bits of "unnecessary" precision to a commentary about the parallelizability of his approach (no comment about the work-efficiency!). Personally, I think that the unum approach is not particularly satisfying as a better alternative to floating point. The advantages that are presented focus on storage space and parallelizability--and the fact that the precision is meant to slowly ("automatically", which I scare-quote because it doesn't appear to be done for the programmer) increase implies that the hardware has to deal with variable-sized data formats, which is not conducive to more efficient computing. It's only efficient to use more compressed formats if it's saving you a lot of work compared to a dense format--and it's not at all clear that unums are actually saving work.
- deepnotderp 10y ago
  > I didn't see a response from Gustafson to the latter accusation--the slide deck I see has him shifting from an argument about the power usage involved in storing 64 bits of "unnecessary" precision to a commentary about the parallelizability of his approach (no comment about the work-efficiency!). Gustafson has addressed this. His argument essentially boils down to the fact that moving 64 bits to memory costs far more than doing the actual computation. Therefore, you have to spend more transistors doing the computation for unums, but you spend much less energy overall, since you have to transport less energy.
dnautics 10y ago
so I'm the author of one of the first usable unum implementations. We're actually moving to something called sigmoid numbers which are even better. The lecture doesn't cover "valid mode" but that's more like the "unum". At the end of the video, there's a demonstration where I show some very interesting results concerning machine learning. https://www.youtube.com/watch?v=aP0Y1uAA-2Y https://www.youtube.com/watch?v=aP0Y1uAA-2Y You can try out sigmoid numbers in julia here: https://github.com/interplanetary-robot/SigmoidNumbers https://github.com/interplanetary-robot/SigmoidNumbers Also, I am working on some verilog implementations, so we may have this in hardware rather soon. A few things about sigmoid numbers: 1) they have fixed sizes, so you don't have to worry about keeping track of things (which was a pain). 2) they have a property I call "isomorphic structure" - which means that zero-padding does not affect the value, and truncation gives you the nearest value at a lower bit resolution. 3) they really do perform better than floating points in several applications. There's a higher bit-per-bit entropy for 'most things you want to do', so the information content is richer. The distribution of the numbers is "more human" - that is to say at a certain point you stop caring about the details around the number, and more care that it's 'really big' or 'really small'; and the distribution reflects that.
- modeless 10y ago
  > very interesting results concerning machine learning This is the big chance for alternative number formats. a whole lot of people are working on neural net ASICs these days and IEEE 754 is not a requirement. A new number format with a low power/low area hardware implementation could easily find adoption in this new area.
  - dnautics 10y ago
    Thanks. I'm working on an Exciting architecture in that direction. We'll see if I can raise money.
- contravariant 10y ago
  Those sigmoid numbers look interesting. Is there an article somewhere that explains how they work?
andrepd 10y ago
You say it's pretty bad and nonsense, yet you haven't found (or even looked very hard) about alternatives. So maybe don't jump to the conclusion that floating point is terrible just yet :)

com2kid 10y ago

Just reading the list of disadvantages, the format sounds interesting but not that practical in a world where massive numbers of FPUs operating within a thermal limit is a primary design concern.

Retra 10y ago

Wikipedia provides a pretty good summary: William Kahan claims the following issues with unums: 1. Unum computation does not always deliver correct results. 2. The description of unums sidesteps using unum for solving calculus problems. 3. Unums can be expensive in terms of time and power consumption. 4. Each computation in unum space is likely to change the bit length of the structure. This requires either unpacking them into a fixed-size space, or data allocation, deallocation, and garbage collection during unum operations, similar to the issues for dealing with variable-length records in mass storage. 5. Unums provide only two kinds of numerical exception, quiet and signaling NaN (Not-a-Number). 6. Unum computation may deliver overly loose bounds from the selection of an algebraically correct but numerically unstable algorithm. 7. The costs and benefits of unum over short precision floating point for problems requiring low precision are not obvious. 8. Solving differential equations and evaluating integrals with unums guarantee correct answers but may not be as fast as methods that usually work. Point #4 makes implementation tricky and slow. Point #6 (from what I understand) means unum operations tend to guarantee correctness by being overly vague - by denoting results that FP can produce using ranges that are just too big to be useful.

gajjanag 10y ago

> Does anyone know more about this? Are there any chips out there that support it? Not an answer to your question, but I found out recently that there is definitely significant interest in ideas like interval arithmetic in hardware among mathematicians. The application I am somewhat familiar with is "brute forcing" proofs of inequalities, which can get very annoying, even for apparently easy stuff like 2 variable ones. Indeed, "The Princeton Companion to Applied Mathematics" edited by N. Higham et al has an entire article on this (in section II.20). In particular, the article has a reference to a library called filib++ (http://www2.math.uni-wuppertal.de/~xsc/software/filib.html http://www2.math.uni-wuppertal.de/~xsc/software/filib.html). I unfortunately have no idea how it performs/how good it is. By the way, thanks a lot for the Unum reference; seems like it is still in too early a stage for inclusion in the book above.

delhanty 10y ago
Floating point is Nick Higham's thing isn't? I have his book "Accuracy and Stability of Numerical Algorithms, Second Edition" http://www.ma.man.ac.uk/~higham/asna/ http://www.ma.man.ac.uk/~higham/asna/ but I haven't got very far with it yet. Maybe the article in the Princeton Companion is a summarized version of his book?
jacobolus 10y ago
I emailed Nick Higham asking about Unums a year and a half ago. Hopefully he doesn’t mind me quoting from private email: > I have Gustafson's book and have read a lot of it. I don't yet know what to make of it. [..] His Unum system is a completely different way of working to floating point, so all I can say is that standard error analysis techniques will probably not be applicable. > The book does not yet seem to have atttracted much attention in the numerical analysis community.

nine_k 10y ago

If NaN is nonsense, what would you have 1/0 or acos(2) to produce? (An exception is the same NaN, only less convenientlying packaged.)

rnhmjoj 10y ago
Ideally I would restrict all functions to their natural domain using refinement types. This ways 1/0 simply doesn't type check and you don't have to bother assigning it some nonsensical value. However I can see something like this becoming much more a hassle than a benefit when things get complicated enough.
- brianberns 10y ago
  I'm not very familiar with refinement types. How would that work? For example, how would you write a function that answers the mean value of an array of numbers? You could define Array.Length to be of type NonNegativeInteger, but that still includes 0. Alternatively, you'd have to define a NonEmptyArray type that excludes arrays of length 0, but that seems less than useful.
  - rnhmjoj 10y ago
    Yes, that's exactly how it works. See this article for an example: http://nikita-volkov.github.io/refined/ http://nikita-volkov.github.io/refined/ The thing is that, instead of redefining everything yourself, you could use a library that provides you with a set of predicates to compose and build the refinement you need. There are also much sophisticated ways to do this, like Liquid Haskell.
  - protomikron 10y ago
    > Alternatively, you'd have to define a NonEmptyArray type that excludes arrays of length 0, but that seems less than useful. I think the usefulness of a specific NonEmptyArray type is controversial, and I would say such types could be extremely useful. AFAIK there have been discussions if Haskell's Prelude (part of the standard library providing a name space for convenient functions and types, similar to the functions you get in e.g. Python like map, zip, filter, range, etc.) should be more save by not advocating too generic types and functions like the standard `map`, `head` and `tail` implementations. Functions like `head` and `tail` can be seen as problematic as they can fail and cause runtime errors (the head and tail of an empty list cause a runtime error), but I do not think they already changed that (or if it's even a topic of discussion at the moment). It's a little bit a matter of taste. Think about Unix' "head" program that by default prints maximal 10 lines of a file. How should we deal with the following? $ touch empty.txt $ head -n1 empty.txt Currently this works - you do not get output and $? == 0, but one could also say "head" should panic here.

jcranmer 10y ago

A cursory examination of the documentation for and against the unum stuff seems to indicate a vehement ad hominem argument running between Gustafson and Kahan, which makes it somewhat hard to find objective information. The unum stuff does seem to be greatly oversold, though. Fundamentally, there is a problem in mathematics: some equations naturally blow up tiny uncertainties into major ones. Indeed, in chaotic systems, such uncertainties can be extremely difficult to describe accurately. Floating point numbers would give you an implausibly precise but meaninglessly inaccurate number, whereas unums would indicate that the answer is basically everything. The difference here basically boils down to Kahan arguing that you should do numerical analysis first to figure out if the result will be meaningful, while Gustafson wants to throw calculus and numerical analysis out the window. Surprisingly, the dispute is less about the actual format than the style of numerical algorithms. Gustafson seems to be advocating an iterative, increasing-precision approach to solving problems like root finding in lieu of standard numerical techniques. Kahan's retort is that the state-of-the-art techniques are more than sufficient, and that Gustafson's techniques take longer to achieve acceptable results. I didn't see a response from Gustafson to the latter accusation--the slide deck I see has him shifting from an argument about the power usage involved in storing 64 bits of "unnecessary" precision to a commentary about the parallelizability of his approach (no comment about the work-efficiency!). Personally, I think that the unum approach is not particularly satisfying as a better alternative to floating point. The advantages that are presented focus on storage space and parallelizability--and the fact that the precision is meant to slowly ("automatically", which I scare-quote because it doesn't appear to be done for the programmer) increase implies that the hardware has to deal with variable-sized data formats, which is not conducive to more efficient computing. It's only efficient to use more compressed formats if it's saving you a lot of work compared to a dense format--and it's not at all clear that unums are actually saving work.

deepnotderp 10y ago
> I didn't see a response from Gustafson to the latter accusation--the slide deck I see has him shifting from an argument about the power usage involved in storing 64 bits of "unnecessary" precision to a commentary about the parallelizability of his approach (no comment about the work-efficiency!). Gustafson has addressed this. His argument essentially boils down to the fact that moving 64 bits to memory costs far more than doing the actual computation. Therefore, you have to spend more transistors doing the computation for unums, but you spend much less energy overall, since you have to transport less energy.

dnautics 10y ago

so I'm the author of one of the first usable unum implementations. We're actually moving to something called sigmoid numbers which are even better. The lecture doesn't cover "valid mode" but that's more like the "unum". At the end of the video, there's a demonstration where I show some very interesting results concerning machine learning. https://www.youtube.com/watch?v=aP0Y1uAA-2Y https://www.youtube.com/watch?v=aP0Y1uAA-2Y You can try out sigmoid numbers in julia here: https://github.com/interplanetary-robot/SigmoidNumbers https://github.com/interplanetary-robot/SigmoidNumbers Also, I am working on some verilog implementations, so we may have this in hardware rather soon. A few things about sigmoid numbers: 1) they have fixed sizes, so you don't have to worry about keeping track of things (which was a pain). 2) they have a property I call "isomorphic structure" - which means that zero-padding does not affect the value, and truncation gives you the nearest value at a lower bit resolution. 3) they really do perform better than floating points in several applications. There's a higher bit-per-bit entropy for 'most things you want to do', so the information content is richer. The distribution of the numbers is "more human" - that is to say at a certain point you stop caring about the details around the number, and more care that it's 'really big' or 'really small'; and the distribution reflects that.

modeless 10y ago
> very interesting results concerning machine learning This is the big chance for alternative number formats. a whole lot of people are working on neural net ASICs these days and IEEE 754 is not a requirement. A new number format with a low power/low area hardware implementation could easily find adoption in this new area.
- dnautics 10y ago
  Thanks. I'm working on an Exciting architecture in that direction. We'll see if I can raise money.
contravariant 10y ago
Those sigmoid numbers look interesting. Is there an article somewhere that explains how they work?

andrepd 10y ago

You say it's pretty bad and nonsense, yet you haven't found (or even looked very hard) about alternatives. So maybe don't jump to the conclusion that floating point is terrible just yet :)