5 ms·

I'm not sure what the etiquette wrt. cross posting comments is, but here goes. On Reddit I divulged into why this doesn't pass my sniff test. This seems to be

by Veedrac 11y ago

I'm not sure what the etiquette wrt. cross posting comments is, but here goes. On Reddit I divulged into why this doesn't pass my sniff test. This seems to be dressed up interval arithmetic with a worse subset of the number line as the bounds. https://www.reddit.com/r/programming/comments/4d08g0/a_radical_approach_to_computation_with_real/d1n07o5 https://www.reddit.com/r/programming/comments/4d08g0/a_radic...

jacobolus 11y ago
Did you read Gustafson’s book? I’m guessing not. Considering Gustafson had a distinguished career in parallel computing, winning top awards from ACM and IEEE, and working as director of an Intel research lab and then as a senior fellow at AMD, he probably knows at least a little bit about the subject. I recommend reading the book. Even if you ultimately decide that they’re an offbeat set of ideas that won’t manage to hit the mainstream, it has very clear exposition and does a nice job explaining the current floating point system with great diagrams. My problem with it is mainly that I think he goes off the rails a little bit in the end when he starts throwing standard analysis techniques out the window and advocating that people solve more problems with brute force methods. This could be useful for some problems but I think he oversells it. Also, unums are incompatible with any of the significant amounts of work that have gone into studying numerical stability, etc., for floating point computations, so it’s hard to judge a priori how many types of scientific computing algorithms might need to change. I think your assumptions about what is slow or fast are off base though. Most of the time spent in modern computers is moving data around, not performing arithmetic per se. Adopting something like unums would require quite significant architecture changes, but with a machine designed for it most computations shouldn’t necessarily end up slower than with standard floating point. [This is a bit of a guess, since these don’t exist, making them hard to benchmark.]
- Veedrac 11y ago
  Indeed I have not read the book. I'm not really inclined to buy it, but I did skim the Amazon preview which seemed not to include any of the interesting sections (aka. chapter 6). > I think your assumptions about what is slow or fast are off base. Large lookup tables are unwieldy. If massive lookup tables worked above 8 bit or so, why wouldn't they be used already? > these computing number systems [...] have a dyadic system wherein every represented number is some multiple of a power of two. For IEEE floats, sure. For Unum IIs, logarithmic number systems or level index number systems, no. Except in the sense that every real number is a multiple of some power of two. It's true that Unum IIs include intervals, but I factored that out of the question when I pointed out it's equivalent to just supporting open and half open intervals without them. And then you do indeed have a subset of the reals. I don't really get why you say you're not representing real numbers. 1 is a real number, and is exactly representable in IEEE floating point. (Technically infinities and NaN mean it's not strictly a subset, but that's pedanticism.)
  - jacobolus 11y ago
    If we’re dealing with numerical problems, “real numbers” are not directly computable. All we have are approximations and (in the case of unums) imprecise bins. 1 is an integer. It’s easy to make a computer representation for integers, especially integers with some finite bound. How about exp(1) or arccos(-1)... suddenly not precisely representable anymore.
    - Veedrac 11y ago
      IEEE floats are exact. That there exist real numbers that aren't IEEE floats is why we say IEEE floats are a subset of real numbers, and why operations on IEEE floats have to round to a representable value.
      - jacobolus 11y ago
        Floats are also a subset of the rational numbers. To be specific, they’re a particular subset of the dyadic rationals with a bounded number of significant figures and a bounded exponent. I think it’s misleading / counterproductive to talk about IEEE floats as subsets of “real numbers”, because e.g. Math.PI is a clearly different thing than the “real number” π (whatever that might be). Calling floats “exact” is also IMO problematic, since in practice every computation using floats involves rounding. (I also think it’s problematic for Gustafson to call the numbers in his system “real numbers”, even if you could in principle build new hardware with the idea to perform arithmetic on any particular (decided before hardware design) small set of real numbers you want.)
        Veedrac 11y ago
        I say subset of the real numbers because other number systems don't restrict themselves to rationals. Logarithmic number systems, for example, almost never express rational numbers. It's true that most floating point computations involve rounding, but there's no contradiction in that. If you think floats are inexact you can get into a lot of confusion about things; you might miss, for example, that `(a + b) / 2` is exact (except on overflow to infinity, or languages that don't enforce floating point correctness) or that Javascript's integers are actually just floats. Considering floats as inexact might lead you to believe they represent ranges of numbers, eg. x ± ½eps, which is confusing. I've seen more damage from this belief than most other myths about floats, at least among experienced programmers.
dnautics 11y ago
>Floating point is >Rather fast to calculate; addition is mostly trivial and multiplication is reasonable... Is that true? Having coded a softfloat I have to respectfully disagree. > So slow the values need to be cached (which works for 8-bit values, but not anything bigger), I think you're misinterpreting the description: The values don't need to be cached, caching is chosen specifically to get O(1) calculation.
- Veedrac 11y ago
  I meant in computational resource terms, on hardware. It's obviously a pain to actually write. > The values don't need to be cached I don't see how addition could be implemented in a semi-reasonable amount of time without table lookup, nor does it seem to be mentioned.
  - dnautics 11y ago
    floating point multiplications expend considerable resources... 64-bit mult on x86 is 3 cycles, IIRC.
    - Veedrac 11y ago
      Ergo "multiplication is reasonable" rather than "very fast". 3 cycles doesn't seem like a ton to me (searching suggests it's actually 5, but that's still not a ton for a 64 bit multiply IMO).
      - trsohmers 11y ago
        Other than mathematical correctness, which was why John started working on unums, one of the coolest things about "Unums 2.0" as shown in this presentation is that all four arithmetic operations (add, subtract, multiply, divide) can take the same time. The 64 bit IEEE FPU my company is developing is able to do add and subtract in a single cycle, a multiply or fused multiply add in 3 cycles, and a divide in 17 cycles... all of which are pretty much the fastest reasonable implementations you can get. To give you an idea of actual time to do those things, doing an add takes around 300 picoseconds, while the multiply is around 3,000 picoseconds. An order of magnitude increase in time to do an operation is a lot when it comes to hardware complexity (and thus associated area and power cost).
stephencanon 11y ago
This is more or less my take as well. Beside the issues that you highlighted, the thing that really doesn't pass the sniff test for me is the completely unsupported claim that with SORNs "Uncertainty grows linearly in general". I simply don't see how this can possibly be the case; interval growth in arithmetic is a straightforward consequence of finite precision, independent of what representation is used. The only way you can avoid it is for representations to grow arbitrarily, which introduces its own set of problems.

by Veedrac 11y ago

jacobolus 11y ago

Did you read Gustafson’s book? I’m guessing not. Considering Gustafson had a distinguished career in parallel computing, winning top awards from ACM and IEEE, and working as director of an Intel research lab and then as a senior fellow at AMD, he probably knows at least a little bit about the subject. I recommend reading the book. Even if you ultimately decide that they’re an offbeat set of ideas that won’t manage to hit the mainstream, it has very clear exposition and does a nice job explaining the current floating point system with great diagrams. My problem with it is mainly that I think he goes off the rails a little bit in the end when he starts throwing standard analysis techniques out the window and advocating that people solve more problems with brute force methods. This could be useful for some problems but I think he oversells it. Also, unums are incompatible with any of the significant amounts of work that have gone into studying numerical stability, etc., for floating point computations, so it’s hard to judge a priori how many types of scientific computing algorithms might need to change. I think your assumptions about what is slow or fast are off base though. Most of the time spent in modern computers is moving data around, not performing arithmetic per se. Adopting something like unums would require quite significant architecture changes, but with a machine designed for it most computations shouldn’t necessarily end up slower than with standard floating point. [This is a bit of a guess, since these don’t exist, making them hard to benchmark.]

Veedrac 11y ago
Indeed I have not read the book. I'm not really inclined to buy it, but I did skim the Amazon preview which seemed not to include any of the interesting sections (aka. chapter 6). > I think your assumptions about what is slow or fast are off base. Large lookup tables are unwieldy. If massive lookup tables worked above 8 bit or so, why wouldn't they be used already? > these computing number systems [...] have a dyadic system wherein every represented number is some multiple of a power of two. For IEEE floats, sure. For Unum IIs, logarithmic number systems or level index number systems, no. Except in the sense that every real number is a multiple of some power of two. It's true that Unum IIs include intervals, but I factored that out of the question when I pointed out it's equivalent to just supporting open and half open intervals without them. And then you do indeed have a subset of the reals. I don't really get why you say you're not representing real numbers. 1 is a real number, and is exactly representable in IEEE floating point. (Technically infinities and NaN mean it's not strictly a subset, but that's pedanticism.)
- jacobolus 11y ago
  If we’re dealing with numerical problems, “real numbers” are not directly computable. All we have are approximations and (in the case of unums) imprecise bins. 1 is an integer. It’s easy to make a computer representation for integers, especially integers with some finite bound. How about exp(1) or arccos(-1)... suddenly not precisely representable anymore.
  - Veedrac 11y ago
    IEEE floats are exact. That there exist real numbers that aren't IEEE floats is why we say IEEE floats are a subset of real numbers, and why operations on IEEE floats have to round to a representable value.
    - jacobolus 11y ago
      Floats are also a subset of the rational numbers. To be specific, they’re a particular subset of the dyadic rationals with a bounded number of significant figures and a bounded exponent. I think it’s misleading / counterproductive to talk about IEEE floats as subsets of “real numbers”, because e.g. Math.PI is a clearly different thing than the “real number” π (whatever that might be). Calling floats “exact” is also IMO problematic, since in practice every computation using floats involves rounding. (I also think it’s problematic for Gustafson to call the numbers in his system “real numbers”, even if you could in principle build new hardware with the idea to perform arithmetic on any particular (decided before hardware design) small set of real numbers you want.)
      - Veedrac 11y ago
        I say subset of the real numbers because other number systems don't restrict themselves to rationals. Logarithmic number systems, for example, almost never express rational numbers. It's true that most floating point computations involve rounding, but there's no contradiction in that. If you think floats are inexact you can get into a lot of confusion about things; you might miss, for example, that `(a + b) / 2` is exact (except on overflow to infinity, or languages that don't enforce floating point correctness) or that Javascript's integers are actually just floats. Considering floats as inexact might lead you to believe they represent ranges of numbers, eg. x ± ½eps, which is confusing. I've seen more damage from this belief than most other myths about floats, at least among experienced programmers.

dnautics 11y ago

>Floating point is >Rather fast to calculate; addition is mostly trivial and multiplication is reasonable... Is that true? Having coded a softfloat I have to respectfully disagree. > So slow the values need to be cached (which works for 8-bit values, but not anything bigger), I think you're misinterpreting the description: The values don't need to be cached, caching is chosen specifically to get O(1) calculation.

Veedrac 11y ago
I meant in computational resource terms, on hardware. It's obviously a pain to actually write. > The values don't need to be cached I don't see how addition could be implemented in a semi-reasonable amount of time without table lookup, nor does it seem to be mentioned.
- dnautics 11y ago
  floating point multiplications expend considerable resources... 64-bit mult on x86 is 3 cycles, IIRC.
  - Veedrac 11y ago
    Ergo "multiplication is reasonable" rather than "very fast". 3 cycles doesn't seem like a ton to me (searching suggests it's actually 5, but that's still not a ton for a 64 bit multiply IMO).
    - trsohmers 11y ago
      Other than mathematical correctness, which was why John started working on unums, one of the coolest things about "Unums 2.0" as shown in this presentation is that all four arithmetic operations (add, subtract, multiply, divide) can take the same time. The 64 bit IEEE FPU my company is developing is able to do add and subtract in a single cycle, a multiply or fused multiply add in 3 cycles, and a divide in 17 cycles... all of which are pretty much the fastest reasonable implementations you can get. To give you an idea of actual time to do those things, doing an add takes around 300 picoseconds, while the multiply is around 3,000 picoseconds. An order of magnitude increase in time to do an operation is a lot when it comes to hardware complexity (and thus associated area and power cost).

stephencanon 11y ago

This is more or less my take as well. Beside the issues that you highlighted, the thing that really doesn't pass the sniff test for me is the completely unsupported claim that with SORNs "Uncertainty grows linearly in general". I simply don't see how this can possibly be the case; interval growth in arithmetic is a straightforward consequence of finite precision, independent of what representation is used. The only way you can avoid it is for representations to grow arbitrarily, which introduces its own set of problems.