8 ms·

Posits, a New Kind of Number, Improves the Math of AI

220 points by cpeterso 4y ago

novaRom 4y ago
"They also found that the improved accuracy didn’t come at the cost of computation time, only a somewhat increased chip area and power consumption." Yes. Nothing for free.
- evdubs 4y ago
  Part of it is not wasting so many bit patterns on valid NaN values. That is sort of free, I think. > Whereas floating point numbers are polluted with NaN values, posits are cleansed of such unclean special values. https://www.cs.cornell.edu/courses/cs6120/2019fa/blog/posits/ https://www.cs.cornell.edu/courses/cs6120/2019fa/blog/posits...
  - js8 4y ago
    I think NaNs are pretty useful. I think we should have reserved 0x80000000 as a NaN value for integers.
    - Aardwolf 4y ago
      On signed integers, -0x80000000 returns 0x80000000 itself since there's no positive value opposite to this negative value, so 0x80000000 as NaN would make sense, but alternatively 0x80000000 as projective infinity would also make sense
    - ljosifov 4y ago
      +1. I do that sometimes, have INT_NAN map to INT_MIN. It is of limited practical use without hardware support such that as in FLOAT_NAN, any operation involving it, results in NAN. Still - I find that improves my code readability, so small +ve gain imho.
    - adgjlsfhk1 4y ago
      posits still have Near which is a single NaN like value. the problem with floats is that they have -0, -inf, inf and a bunch of NaN values. float16 for example wastes over 1000 of the 65000 numbers or can represent on non-finite numbers.
    - IshKebab 4y ago
      That's a really interesting idea. On the other hand it does add quite a lot of complexity to something that is quite simple. And it raises some awkward questions like how do you represent NaN for unsigned integers and if you don't then you have the awkward fact that there's one more unsigned than signed number. I can imagine an alternative timeline where it worked like that though.
- vintermann 4y ago
  Of course nothing for free, but the point is to trade things you don't need for things you do. I also expect that chip area and power consumption for IEEE floating point has been optimized quite a bit more than for this novel number format, so there may be some efficiency left on the table on those particular metrics.
- bXVsbGVy 4y ago
  The posit is a compressed floating point encoding with a flexible data structure with a trade-off of precision (the number of bits after the dot) and magnitude (the absolute value). It uses more hardware because the posit is decoded into a floating point larger than a equivalent IEEE754 with the same number of bits. So, the logic units needs to be larger.
MikeDelta 4y ago
I thought this was about different numbers like rational and complex, but it is about number representation (of floats), see also https://en.wikipedia.org/wiki/Unum_(number_format)#Unum_III https://en.wikipedia.org/wiki/Unum_(number_format)#Unum_III
- parasti 4y ago
  Unfortunately the Wikipedia article is extremely light on details of the actual format.
  - zeotroph 4y ago
    The(?) posit website has a pdf[1] which goes into more details under "3.3 Posit format encoding". It also mentions a "quire" (and the footnote for the quire value is self referential?) , can anyone explain that concept? 1: https://posithub.org/docs/posit_standard-2.pdf https://posithub.org/docs/posit_standard-2.pdf
    - SanjayMehta 4y ago
      As I understood it, it's a large internal register to prevent fractional errors from accumulating.
      - josefx 4y ago
        Wasn't that a gigantic issue with x87 floating point implementation? It would internally use 80 bit registers and the result of any computation was completely at the mercy of how the compiler used those 80 bit registers as every move to system memory would drop precision back to the 32 bit or 64 bit specified by the program.
        SanjayMehta 4y ago
        I don't really know anything about that, but it appears here that all this is done in hardware without any compilers involved.
dr_dshiv 4y ago
Beautiful idea: a number representation where the distribution of accuracy aligns with the distribution of typical use. >Real numbers can’t be perfectly represented in hardware simply because there are infinitely many of them. To fit into a designated number of bits, many real numbers have to be rounded. The advantage of posits comes from the way the numbers they represent exactly are distributed along the number line. In the middle of the number line, around 1 and -1, there are more posit representations than floating point. And at the wings, going out to large negative and positive numbers, posit accuracy falls off more gracefully than floating point. >“It’s a better match for the natural distribution of numbers in a calculation,” says Gustafson. “It’s the right dynamic range, and it’s the right accuracy where you need more accuracy. There’s an awful lot of bit patterns in floating-point arithmetic no one ever uses. And that’s waste.”
- boberoni 4y ago
  I like this idea of choosing a representation for numbers based on the use case. Like, "ints" and "floats" but even more specialized. For example, what would be the most efficient binary representation of probabilities values between 0 and 1? In the future, I can imagine hardware + software that is specialized for examples like this.
  - dr_dshiv 4y ago
    Makes me wonder about quantum computing possibilities.
  - js8 4y ago
    I feel like probabilities are best stored as inverse logistic function, because the closer the probability is to 0 or 1, the less you care about significant digits, because the events are going (not) to happen anyway. (That's why I am proponent of LNS.)
    - jbay808 4y ago
      I'd agree with this suggestion. Furthermore this does not need a new number format, as IEEE floating point is already pretty ideally distributed. https://en.wikipedia.org/wiki/Logit https://en.wikipedia.org/wiki/Logit
vintermann 4y ago
Novel number representations are cool. I came across Paul Tarau's "hereditarily binary natural numbers" a few years ago, and via them, Knuth's earlier TCALC representation (one of the many small side projects Knuth has done over the years, which for some reason hasn't gotten much attention). They go pretty much the other way than this representation: they allow huge numbers, much larger than can be represented in regular notation (incl. floating point) to be represented and calculated with efficiently and exactly. Tarau's system improves slightly on Knuth's in that representations are unique. He also proves that they at worst require twice as many bits as regular representation. Knuth's and Tarau's system are variable-length and limited to naturals, but it seems like it would be easy enough to extend them to rationals and fix the representation length.
black_puppydog 4y ago
Kinda burying the lede there... The actual paper [0] is called "PERCIVAL: Open-Source Posit RISC-V Core With Quire Capability". :) [0]: https://ieeexplore.ieee.org/document/9817027/ https://ieeexplore.ieee.org/document/9817027/
freemint 4y ago
I kinda wish Gustafson would focus on something more productive but he is still trying to boil the ocean.
js8 4y ago
I don't understand why don't we just use https://en.wikipedia.org/wiki/Logarithmic_number_system https://en.wikipedia.org/wiki/Logarithmic_number_system. It seems much simpler than the posit proposal and it has the same advantage (most precision around 1 and -1).
- hakuseki 4y ago
  A logarithmic representation is great for multiplication, but adding numbers becomes an expensive operation.
  - js8 4y ago
    Today, both integer addition and multiplication is IMHO done in similar speeds in silicon, although multiplication circuits are way larger. In the logarithmic representation, multiplication becomes addition. I don't think that just by doing everything in logarithms you really increase the overall complexity, rather you transfer it from multiplication onto addition, so I don't think adding logarithms should be more expensive than integer multiplication.. Should it then matter for neural networks, which IIRC require similar amount of additions and multiplications? Even floating-point is actually trading off the simplicity of addition in order to make multiplication easier, because they are quasi-logarithmic representation. However, I wonder if there is a numeric representation where the circuits for addition and multiplication are of similar complexity. Something like half-logarithm, which if applied twice, you get the logarithm.
    - NohatCoder 4y ago
      Addition in logarithmic space is expensive, we don't have any neat ways of doing it. Options include: * Converting back and forth to linear space, using exp and log functions, this is much slower than a regular multiplication. * Evaluating a polynomial that approximates the logarithmic space add, this takes several multiplications, so also much slower. In general we tend to use more additions than multiplications, so trading addition speed for multiplication speed is rarely a good idea, even 1 for 1. If we need a lot of exponentiation keeping some values in logarithmic space may be beneficial, but it has to be those rare cases only.
eurasiantiger 4y ago
I like the concept, but hate the name. There is no semantic connection to what it represents, unlike with float, for example.
- xeonmc 4y ago
  floats = floating decimal point (or perhaps it should be called binary points) Posits have floating range as well as floating precision, so doubly floaty. Therefore I propose that they should be called "Lofties".
- throwawaymaths 4y ago
  The name was created so someday someone might use it for machine learning tasks (perceptrons) and thereby finally create positronic brains.
- bXVsbGVy 4y ago
  Something like "compressed floats" would express well the name.
vitus 4y ago
Hm. The article links to the posit paper [0]: > A posit processing unit takes less circuitry than an IEEE float FPU. With lower power use and smaller silicon footprint, the posit operations per second (POPS) supported by a chip can be significantly higher than the FLOPS using similar hardware resources. The argument seems to be predicated on the cost associated with NaN-handling. The exposition comes across as somewhat arrogant, IMO: > If a programmer finds the need for NaN values, it indicates the program is not yet finished, and the use of valids should be invoked as a sort of numerical debugging environment to find and eliminate possible sources of such outputs. Meanwhile, from the article where people actually tried implementing this thing on an FPGA: > They also found that the improved accuracy didn’t come at the cost of computation time, only a somewhat increased chip area and power consumption. I wonder where the mismatch comes from. [0] http://www.johngustafson.net/pdfs/BeatingFloatingPoint.pdf http://www.johngustafson.net/pdfs/BeatingFloatingPoint.pdf
- NohatCoder 4y ago
  I think Gustafson may be the kind of liar that doesn't know he is lying. He knows saying that his design improves upon power and die size sounds good, so he says it.
  - throwawaymaths 4y ago
    Nope, I have seen it tested on an fpga, many years ago. The reference was Berkeley hardfloat. This might not be the best implementation ever, but it is the most accessible open source one.
    - NohatCoder 4y ago
      Where do you save transistors? With the Posit you have to be able to deal with larger mantissa, multiplier size scales with mantissa bits squared, so even a small increase makes quite a mark. Maybe you save a bit by not having denormals, but then parsing the packed float is a bit more complicated in that the bits do not have a fixed division between exponent and mantissa. It is possible that the Posit circuit was smaller by leaving out some feature, like exact rounding, which is quite expensive, but then it is not an apples-to-apples comparison.
thuum7 4y ago
Simplest description by Gustafson https://posithub.org/conga/2019/docs/13/1430-John-Introductory.pdf https://posithub.org/conga/2019/docs/13/1430-John-Introducto...
loicd 4y ago
I don't know about regime notation, but it is nice to see a new format with the most annoying ideas from IEEE754 removed: - Two's complement instead of one's complement - No infinities, no signed zeroes - Only one exceptional value with the same encoding as the smallest signed integer: 10...0 The above means that comparisons work exactly like for signed integers (with the unique exceptional value behaving like -∞). Also, one can test whether x is invalid using `x == -x && x != 0` rather than the ugly `x != x`, i.e. no need to break reflexivity. Even if posits do nothing but remove the one's complement dust off IEEE754, that would be a positive change in my view.
- phkahler 4y ago
  There are a whole bunch of people who insist that lots of the "baggage" of IEEE 754 is really important and must not be abandoned. For most applications it is just extra garbage and for the few there are other ways to do what they want.
  - NohatCoder 4y ago
    That may be true, but you don't need a fundamentally different number representation in order to throw away most of it. Also, the die size cost is quite small. The biggest saving you could make is probably foregoing exactly rounded results, and only stipulate that the result has to be within ±1 lsb of the true value. That would save multipliers from computing a bunch of bits that don't end up in the result anyway, except for the rare case where they decide the rounding. That would probably be a good trade-off for most AI chips. For general purpose CPUs I don't think it is worth the breakage.
  - noobermin 4y ago
    We literally had a post about "old not being broken" on the front page yesterday with a significant discussion on developer churn and now people are already clamoring on other pages to deprecate floating points. Let's have a little memory to shake us out of our habits, shall we?
- throwawaymaths 4y ago
  The big problem with posits is that its relative error depends on its value, this is terrible for a lot of engineering work and scientific simulations where you need to present an error estimate that includes computational error (for ML it's probably fine)
Kalanos 4y ago
"four-order-of-magnitude improvement in the accuracy of matrix multiplication"
- derbOac 4y ago
  Wouldn't this depend on what's in the matrices? I've done work with big numbers and this format screams problems. I can see how it would be great depending on the domain, and wish there was more diversity out there in the wild in this area but this seems a little hyped? Maybe I'm missing something. I'm not saying this doesn't seem useful, just that the article seems to be presenting it as a replacement for floats, when it might be better thought of as another option?
sweezyjeezy 4y ago
One particularly stark 'issue' I've noticed with floating point values in AI (which posits would seem to help with) is the difference between 0 and 1. In binary classification problems we generally assign 1 to one subset of our data, and 0 to its complement - there is not necessarily an inherent asymmetry here and the problem will often not meaningfully change if you switch labels 0<->1. But... if we take for example float16, the smallest model prediction you can represent greater than 0 is 2^{−24} = 5.96*10E−8, but the largest you can represent _less_ than 1 is (binary) 0.111... = 1-2^{-10} = 1-9.77*10E-4. So values around 1 are about 4 orders of magnitude more quantized than those around 0. I don't know if that's necessarily a 'problem' but I have noticed this fact when looking at model predictions.
- NohatCoder 4y ago
  That is fundamentally the same for Posits. If you want constant quantization you should use fixed point, or you could switch your inputs to be +1/-1 for symmetry in this specific case, but I doubt that this is much of a practical issue.
infogulch 4y ago
Posits are pretty damn cool. The posit standard was ratified in March this year. It's only 12 pages and shockingly comprehensible for a standard document: https://posithub.org/docs/posit_standard-2.pdf https://posithub.org/docs/posit_standard-2.pdf Also in March Gustafson hosted the Conference for Next Generation Arithmetic (CoNGA) 2022 sorta 'inside' the Supercomputing Asia (SCA2022) conference. They covered a bunch of different areas not exclusively related to posits. Lots of cool topics including hardware implementations and optimizations, comparisons between different number formats, and a talk and panel from the posit committee about writing the standard and what's next for posits. The session videos are somewhat difficult to find so I made a playlist: https://youtube.com/playlist?list=PLBH9oUUfaYoQNl6-zr_ScsMp4ZFQrG9TE https://youtube.com/playlist?list=PLBH9oUUfaYoQNl6-zr_ScsMp4... (Note the descriptions have talk titles and authors.) The best video intro to posits is probably the one in 2019: https://www.youtube.com/watch?v=JJgT-YphE1Y https://www.youtube.com/watch?v=JJgT-YphE1Y
- infogulch 4y ago
  Probably my favorite CoNGA2022 talk is the first one, A Case for Correctly Rounded Elementary Functions (starts at about 10m), which presents a methodology and library, already implemented into LLVM for floats, of elementary functions (lg, sin, cos, etc etc) that are 1. correctly rounded to the last bit, 2. are faster than the previous algorithms, 3. and can be downcasted to any smaller format while preserving correct rounding. The posit standard requires correctly rounded elementary functions in order for an implementation to be considered compliant. This means that every standards-compliant posit implementation is now also deterministic.
gwbas1c 4y ago
I remember going to an AI meetup where the guy presenting was proposing a totally new computing architecture. I quickly suspected he wasn't very competent about how computers worked, and tried to probe a bit. I suspected he was mentally ill but could "talk the talk" enough to convince people his ideas had merit. But this: A more efficient way of handling floating-point numbers, is probably something worthwhile. I wonder how hard it will be to recompile existing software to take advantage of posits?
- 7373737373 4y ago
  https://en.wikipedia.org/wiki/Outsider_art https://en.wikipedia.org/wiki/Outsider_art
  - gwbas1c 4y ago
    Well, art is art. It doesn't matter who's making it, what their background is, or if they follow preconceived rules. That doesn't apply to engineering because it either works or it doesn't. The presenter misunderstood basic concepts of information. If you don't understand those, you can't design a working computer.
peter_d_sherman 4y ago
>"With their new hardware implementation, which was synthesized in a field-programmable gate array (FPGA), the Complutense team was able to compare computations done using 32-bit floats and 32-bit posits side by side. They assessed their accuracy by comparing them to results using the much more accurate but computationally costly 64-bit floating-point format. Posits showed an astounding four-order-of-magnitude improvement in the accuracy of matrix multiplication" This is nothing short of amazing! I can just imagine future GPUs on FPGA's using Posits rather than (now old!) IEEE floating point formats... (Also, on a probably unrelated note -- might Posits be where we get the future equivalent of Star Trek TNG's character Mr. Data's Positronic (Posit-ronic!) -- AI "brain" from? ???)

220 points by cpeterso 4y ago

"They also found that the improved accuracy didn’t come at the cost of computation time, only a somewhat increased chip area and power consumption." Yes. Nothing for free.

evdubs 4y ago
Part of it is not wasting so many bit patterns on valid NaN values. That is sort of free, I think. > Whereas floating point numbers are polluted with NaN values, posits are cleansed of such unclean special values. https://www.cs.cornell.edu/courses/cs6120/2019fa/blog/posits/ https://www.cs.cornell.edu/courses/cs6120/2019fa/blog/posits...
- js8 4y ago
  I think NaNs are pretty useful. I think we should have reserved 0x80000000 as a NaN value for integers.
  - Aardwolf 4y ago
    On signed integers, -0x80000000 returns 0x80000000 itself since there's no positive value opposite to this negative value, so 0x80000000 as NaN would make sense, but alternatively 0x80000000 as projective infinity would also make sense
  - ljosifov 4y ago
    +1. I do that sometimes, have INT_NAN map to INT_MIN. It is of limited practical use without hardware support such that as in FLOAT_NAN, any operation involving it, results in NAN. Still - I find that improves my code readability, so small +ve gain imho.
  - adgjlsfhk1 4y ago
    posits still have Near which is a single NaN like value. the problem with floats is that they have -0, -inf, inf and a bunch of NaN values. float16 for example wastes over 1000 of the 65000 numbers or can represent on non-finite numbers.
  - IshKebab 4y ago
    That's a really interesting idea. On the other hand it does add quite a lot of complexity to something that is quite simple. And it raises some awkward questions like how do you represent NaN for unsigned integers and if you don't then you have the awkward fact that there's one more unsigned than signed number. I can imagine an alternative timeline where it worked like that though.
vintermann 4y ago
Of course nothing for free, but the point is to trade things you don't need for things you do. I also expect that chip area and power consumption for IEEE floating point has been optimized quite a bit more than for this novel number format, so there may be some efficiency left on the table on those particular metrics.
bXVsbGVy 4y ago
The posit is a compressed floating point encoding with a flexible data structure with a trade-off of precision (the number of bits after the dot) and magnitude (the absolute value). It uses more hardware because the posit is decoded into a floating point larger than a equivalent IEEE754 with the same number of bits. So, the logic units needs to be larger.

MikeDelta 4y ago

I thought this was about different numbers like rational and complex, but it is about number representation (of floats), see also https://en.wikipedia.org/wiki/Unum_(number_format)#Unum_III https://en.wikipedia.org/wiki/Unum_(number_format)#Unum_III

parasti 4y ago
Unfortunately the Wikipedia article is extremely light on details of the actual format.
- zeotroph 4y ago
  The(?) posit website has a pdf[1] which goes into more details under "3.3 Posit format encoding". It also mentions a "quire" (and the footnote for the quire value is self referential?) , can anyone explain that concept? 1: https://posithub.org/docs/posit_standard-2.pdf https://posithub.org/docs/posit_standard-2.pdf
  - SanjayMehta 4y ago
    As I understood it, it's a large internal register to prevent fractional errors from accumulating.
    - josefx 4y ago
      Wasn't that a gigantic issue with x87 floating point implementation? It would internally use 80 bit registers and the result of any computation was completely at the mercy of how the compiler used those 80 bit registers as every move to system memory would drop precision back to the 32 bit or 64 bit specified by the program.
      - SanjayMehta 4y ago
        I don't really know anything about that, but it appears here that all this is done in hardware without any compilers involved.

dr_dshiv 4y ago

Beautiful idea: a number representation where the distribution of accuracy aligns with the distribution of typical use. >Real numbers can’t be perfectly represented in hardware simply because there are infinitely many of them. To fit into a designated number of bits, many real numbers have to be rounded. The advantage of posits comes from the way the numbers they represent exactly are distributed along the number line. In the middle of the number line, around 1 and -1, there are more posit representations than floating point. And at the wings, going out to large negative and positive numbers, posit accuracy falls off more gracefully than floating point. >“It’s a better match for the natural distribution of numbers in a calculation,” says Gustafson. “It’s the right dynamic range, and it’s the right accuracy where you need more accuracy. There’s an awful lot of bit patterns in floating-point arithmetic no one ever uses. And that’s waste.”

boberoni 4y ago
I like this idea of choosing a representation for numbers based on the use case. Like, "ints" and "floats" but even more specialized. For example, what would be the most efficient binary representation of probabilities values between 0 and 1? In the future, I can imagine hardware + software that is specialized for examples like this.
- dr_dshiv 4y ago
  Makes me wonder about quantum computing possibilities.
- js8 4y ago
  I feel like probabilities are best stored as inverse logistic function, because the closer the probability is to 0 or 1, the less you care about significant digits, because the events are going (not) to happen anyway. (That's why I am proponent of LNS.)
  - jbay808 4y ago
    I'd agree with this suggestion. Furthermore this does not need a new number format, as IEEE floating point is already pretty ideally distributed. https://en.wikipedia.org/wiki/Logit https://en.wikipedia.org/wiki/Logit

vintermann 4y ago

Novel number representations are cool. I came across Paul Tarau's "hereditarily binary natural numbers" a few years ago, and via them, Knuth's earlier TCALC representation (one of the many small side projects Knuth has done over the years, which for some reason hasn't gotten much attention). They go pretty much the other way than this representation: they allow huge numbers, much larger than can be represented in regular notation (incl. floating point) to be represented and calculated with efficiently and exactly. Tarau's system improves slightly on Knuth's in that representations are unique. He also proves that they at worst require twice as many bits as regular representation. Knuth's and Tarau's system are variable-length and limited to naturals, but it seems like it would be easy enough to extend them to rationals and fix the representation length.

black_puppydog 4y ago

Kinda burying the lede there... The actual paper [0] is called "PERCIVAL: Open-Source Posit RISC-V Core With Quire Capability". :) [0]: https://ieeexplore.ieee.org/document/9817027/ https://ieeexplore.ieee.org/document/9817027/

freemint 4y ago

I kinda wish Gustafson would focus on something more productive but he is still trying to boil the ocean.

js8 4y ago

I don't understand why don't we just use https://en.wikipedia.org/wiki/Logarithmic_number_system https://en.wikipedia.org/wiki/Logarithmic_number_system. It seems much simpler than the posit proposal and it has the same advantage (most precision around 1 and -1).

hakuseki 4y ago
A logarithmic representation is great for multiplication, but adding numbers becomes an expensive operation.
- js8 4y ago
  Today, both integer addition and multiplication is IMHO done in similar speeds in silicon, although multiplication circuits are way larger. In the logarithmic representation, multiplication becomes addition. I don't think that just by doing everything in logarithms you really increase the overall complexity, rather you transfer it from multiplication onto addition, so I don't think adding logarithms should be more expensive than integer multiplication.. Should it then matter for neural networks, which IIRC require similar amount of additions and multiplications? Even floating-point is actually trading off the simplicity of addition in order to make multiplication easier, because they are quasi-logarithmic representation. However, I wonder if there is a numeric representation where the circuits for addition and multiplication are of similar complexity. Something like half-logarithm, which if applied twice, you get the logarithm.
  - NohatCoder 4y ago
    Addition in logarithmic space is expensive, we don't have any neat ways of doing it. Options include: * Converting back and forth to linear space, using exp and log functions, this is much slower than a regular multiplication. * Evaluating a polynomial that approximates the logarithmic space add, this takes several multiplications, so also much slower. In general we tend to use more additions than multiplications, so trading addition speed for multiplication speed is rarely a good idea, even 1 for 1. If we need a lot of exponentiation keeping some values in logarithmic space may be beneficial, but it has to be those rare cases only.

eurasiantiger 4y ago

I like the concept, but hate the name. There is no semantic connection to what it represents, unlike with float, for example.

xeonmc 4y ago
floats = floating decimal point (or perhaps it should be called binary points) Posits have floating range as well as floating precision, so doubly floaty. Therefore I propose that they should be called "Lofties".
throwawaymaths 4y ago
The name was created so someday someone might use it for machine learning tasks (perceptrons) and thereby finally create positronic brains.
bXVsbGVy 4y ago
Something like "compressed floats" would express well the name.

vitus 4y ago

Hm. The article links to the posit paper [0]: > A posit processing unit takes less circuitry than an IEEE float FPU. With lower power use and smaller silicon footprint, the posit operations per second (POPS) supported by a chip can be significantly higher than the FLOPS using similar hardware resources. The argument seems to be predicated on the cost associated with NaN-handling. The exposition comes across as somewhat arrogant, IMO: > If a programmer finds the need for NaN values, it indicates the program is not yet finished, and the use of valids should be invoked as a sort of numerical debugging environment to find and eliminate possible sources of such outputs. Meanwhile, from the article where people actually tried implementing this thing on an FPGA: > They also found that the improved accuracy didn’t come at the cost of computation time, only a somewhat increased chip area and power consumption. I wonder where the mismatch comes from. [0] http://www.johngustafson.net/pdfs/BeatingFloatingPoint.pdf http://www.johngustafson.net/pdfs/BeatingFloatingPoint.pdf

NohatCoder 4y ago
I think Gustafson may be the kind of liar that doesn't know he is lying. He knows saying that his design improves upon power and die size sounds good, so he says it.
- throwawaymaths 4y ago
  Nope, I have seen it tested on an fpga, many years ago. The reference was Berkeley hardfloat. This might not be the best implementation ever, but it is the most accessible open source one.
  - NohatCoder 4y ago
    Where do you save transistors? With the Posit you have to be able to deal with larger mantissa, multiplier size scales with mantissa bits squared, so even a small increase makes quite a mark. Maybe you save a bit by not having denormals, but then parsing the packed float is a bit more complicated in that the bits do not have a fixed division between exponent and mantissa. It is possible that the Posit circuit was smaller by leaving out some feature, like exact rounding, which is quite expensive, but then it is not an apples-to-apples comparison.

thuum7 4y ago

Simplest description by Gustafson https://posithub.org/conga/2019/docs/13/1430-John-Introductory.pdf https://posithub.org/conga/2019/docs/13/1430-John-Introducto...

loicd 4y ago

I don't know about regime notation, but it is nice to see a new format with the most annoying ideas from IEEE754 removed: - Two's complement instead of one's complement - No infinities, no signed zeroes - Only one exceptional value with the same encoding as the smallest signed integer: 10...0 The above means that comparisons work exactly like for signed integers (with the unique exceptional value behaving like -∞). Also, one can test whether x is invalid using `x == -x && x != 0` rather than the ugly `x != x`, i.e. no need to break reflexivity. Even if posits do nothing but remove the one's complement dust off IEEE754, that would be a positive change in my view.

phkahler 4y ago
There are a whole bunch of people who insist that lots of the "baggage" of IEEE 754 is really important and must not be abandoned. For most applications it is just extra garbage and for the few there are other ways to do what they want.
- NohatCoder 4y ago
  That may be true, but you don't need a fundamentally different number representation in order to throw away most of it. Also, the die size cost is quite small. The biggest saving you could make is probably foregoing exactly rounded results, and only stipulate that the result has to be within ±1 lsb of the true value. That would save multipliers from computing a bunch of bits that don't end up in the result anyway, except for the rare case where they decide the rounding. That would probably be a good trade-off for most AI chips. For general purpose CPUs I don't think it is worth the breakage.
- noobermin 4y ago
  We literally had a post about "old not being broken" on the front page yesterday with a significant discussion on developer churn and now people are already clamoring on other pages to deprecate floating points. Let's have a little memory to shake us out of our habits, shall we?
throwawaymaths 4y ago
The big problem with posits is that its relative error depends on its value, this is terrible for a lot of engineering work and scientific simulations where you need to present an error estimate that includes computational error (for ML it's probably fine)

Kalanos 4y ago

"four-order-of-magnitude improvement in the accuracy of matrix multiplication"

derbOac 4y ago
Wouldn't this depend on what's in the matrices? I've done work with big numbers and this format screams problems. I can see how it would be great depending on the domain, and wish there was more diversity out there in the wild in this area but this seems a little hyped? Maybe I'm missing something. I'm not saying this doesn't seem useful, just that the article seems to be presenting it as a replacement for floats, when it might be better thought of as another option?

sweezyjeezy 4y ago

One particularly stark 'issue' I've noticed with floating point values in AI (which posits would seem to help with) is the difference between 0 and 1. In binary classification problems we generally assign 1 to one subset of our data, and 0 to its complement - there is not necessarily an inherent asymmetry here and the problem will often not meaningfully change if you switch labels 0<->1. But... if we take for example float16, the smallest model prediction you can represent greater than 0 is 2^{−24} = 5.96*10E−8, but the largest you can represent _less_ than 1 is (binary) 0.111... = 1-2^{-10} = 1-9.77*10E-4. So values around 1 are about 4 orders of magnitude more quantized than those around 0. I don't know if that's necessarily a 'problem' but I have noticed this fact when looking at model predictions.

NohatCoder 4y ago
That is fundamentally the same for Posits. If you want constant quantization you should use fixed point, or you could switch your inputs to be +1/-1 for symmetry in this specific case, but I doubt that this is much of a practical issue.

infogulch 4y ago

Posits are pretty damn cool. The posit standard was ratified in March this year. It's only 12 pages and shockingly comprehensible for a standard document: https://posithub.org/docs/posit_standard-2.pdf https://posithub.org/docs/posit_standard-2.pdf Also in March Gustafson hosted the Conference for Next Generation Arithmetic (CoNGA) 2022 sorta 'inside' the Supercomputing Asia (SCA2022) conference. They covered a bunch of different areas not exclusively related to posits. Lots of cool topics including hardware implementations and optimizations, comparisons between different number formats, and a talk and panel from the posit committee about writing the standard and what's next for posits. The session videos are somewhat difficult to find so I made a playlist: https://youtube.com/playlist?list=PLBH9oUUfaYoQNl6-zr_ScsMp4ZFQrG9TE https://youtube.com/playlist?list=PLBH9oUUfaYoQNl6-zr_ScsMp4... (Note the descriptions have talk titles and authors.) The best video intro to posits is probably the one in 2019: https://www.youtube.com/watch?v=JJgT-YphE1Y https://www.youtube.com/watch?v=JJgT-YphE1Y

infogulch 4y ago
Probably my favorite CoNGA2022 talk is the first one, A Case for Correctly Rounded Elementary Functions (starts at about 10m), which presents a methodology and library, already implemented into LLVM for floats, of elementary functions (lg, sin, cos, etc etc) that are 1. correctly rounded to the last bit, 2. are faster than the previous algorithms, 3. and can be downcasted to any smaller format while preserving correct rounding. The posit standard requires correctly rounded elementary functions in order for an implementation to be considered compliant. This means that every standards-compliant posit implementation is now also deterministic.

gwbas1c 4y ago

I remember going to an AI meetup where the guy presenting was proposing a totally new computing architecture. I quickly suspected he wasn't very competent about how computers worked, and tried to probe a bit. I suspected he was mentally ill but could "talk the talk" enough to convince people his ideas had merit. But this: A more efficient way of handling floating-point numbers, is probably something worthwhile. I wonder how hard it will be to recompile existing software to take advantage of posits?

7373737373 4y ago
https://en.wikipedia.org/wiki/Outsider_art https://en.wikipedia.org/wiki/Outsider_art
- gwbas1c 4y ago
  Well, art is art. It doesn't matter who's making it, what their background is, or if they follow preconceived rules. That doesn't apply to engineering because it either works or it doesn't. The presenter misunderstood basic concepts of information. If you don't understand those, you can't design a working computer.

peter_d_sherman 4y ago

>"With their new hardware implementation, which was synthesized in a field-programmable gate array (FPGA), the Complutense team was able to compare computations done using 32-bit floats and 32-bit posits side by side. They assessed their accuracy by comparing them to results using the much more accurate but computationally costly 64-bit floating-point format. Posits showed an astounding four-order-of-magnitude improvement in the accuracy of matrix multiplication" This is nothing short of amazing! I can just imagine future GPUs on FPGA's using Posits rather than (now old!) IEEE floating point formats... (Also, on a probably unrelated note -- might Posits be where we get the future equivalent of Star Trek TNG's character Mr. Data's Positronic (Posit-ronic!) -- AI "brain" from? ???)