4 ms·

One of the creators of posits here (I came up with the name and i think ES is my idea, did the first full soft versions in julia, and designed the first circuit

One of the creators of posits here (I came up with the name and i think ES is my idea, did the first full soft versions in julia, and designed the first circuits, including a cool optimization for addition). my personal stance is that posits are not great for scientific work precisely because of the difficulties with actually solving error propagation. Hopefully i can give a bit more measured insights into why the "parlor tricks" appear in the posits context. John's background is in scientific compute/HPC and he previously advocated for using unums (which do fully track errors) and there is a version of posits (called valids) which do track errors, encouraging the user to combine with other techniques to cut the error bounds using invariants, but that requires algorithmic shift. Alas, a lot of examples were lifted from the unums book and sort of square peg/round holed into posits. you can see an example of algorithmic shift in the demo of matrix multiplication in the stanford talk (that demo is me; linked in OP). as for me, i was much more interested in lower bit representation for ml applications where you ~don't care about error propagation. this also appears in the talk. as it wound up, Facebook took some interest in it for AI but they nih'd it and redid the mantissa as logarithmic (which i think was a mistake). and anyway redoing your silicon it turns out to be a pain in the ass (quires only make sense in the burn-the-existing-world perspective and are not so bad for training pipelines, where iirc kronecker product dominates), but the addition operation takes up quite a bit more floorspace, and just quantizing to int4 is with grouped scaling factors is easier with existing gpu pipelines, even custom hardware too. fun side fact: Positron.ai, was so-named on the off chance that using posits makes sense (you can see the through line to science fiction that i was attempting to manifest when i came up with the name)

dnautics 1y ago
turns out only the slides are linked in op. here is the live recording: https://youtu.be/aP0Y1uAA-2Y?feature=shared https://youtu.be/aP0Y1uAA-2Y?feature=shared
andrepd 1y ago
> and designed the first circuits, including a cool optimization for addition Curious, what trick? :) Wishing for mainstream CPU support for anything but IEEE numbers was always a pipe dream on anything but the decades-long term, but I gotta be honest, I was hoping the current AI hype wave would bring some custom silicon for alternative float formats, Posits included. > the addition operation takes up quite a bit more floorspace, and just quantizing to int4 is with grouped scaling factors is easier with existing gpu pipelines Can you elaborate on this part?
- dnautics 1y ago
  > trick Normally fp numbers have an "invisible leading one bit", so an ieee fp is (1+0.nnnn) x 2^E where nnn is the mantissa and E is the exponent. as posits are always two's complement, you can model negative posits as having an "invisible leading two bit", aka (2-0.nnnn) x 2^E-1. this greatly simplifies circuitry for addition and multiplication. > the addition operation takes up quite a bit more floorspace, I miswrote, it's multiplication. This is simple: since the math requires floorspace of size n^2 where n is the maximum resolution of the mantissa, and posits have more resolution around unity. we did floorspace estimates for fp32 versus p32 and it looked like the extra space required for the mantissa did not overtake the savings from the crazy logic required for sign flipping, nans, and denormals that IEEE requires that posits don't. > I was hoping the current AI hype wave would bring some custom silicon for alternative float formats, Posits included i was hoping for this too!
  - adgjlsfhk1 1y ago
    I really want a standards body to standardize a positized version of floating point numbers. Everything about the encoding of posits (i.e. NaR, no -0.0, exponent encoding) is so much nicer. The tapered precision part of posits is IMO the least interesting part.
    - dnautics 1y ago
      the twos complement thing is really elegant (-p == (posit -(int p))) in c notation
      - adgjlsfhk1 1y ago
        relatedly, having the same ordering as signed ints just makes everything so much easier. You can just use your integer radix sort to sort your floats without having to deal with sign and nans first!
        immibis 1y ago
gsf_emergency_2 1y ago
2018 blog post on the FB effort: https://engineering.fb.com/2018/11/08/ai-research/floating-point-math/ https://engineering.fb.com/2018/11/08/ai-research/floating-p...

dnautics 1y ago

turns out only the slides are linked in op. here is the live recording: https://youtu.be/aP0Y1uAA-2Y?feature=shared https://youtu.be/aP0Y1uAA-2Y?feature=shared

andrepd 1y ago

> and designed the first circuits, including a cool optimization for addition Curious, what trick? :) Wishing for mainstream CPU support for anything but IEEE numbers was always a pipe dream on anything but the decades-long term, but I gotta be honest, I was hoping the current AI hype wave would bring some custom silicon for alternative float formats, Posits included. > the addition operation takes up quite a bit more floorspace, and just quantizing to int4 is with grouped scaling factors is easier with existing gpu pipelines Can you elaborate on this part?

dnautics 1y ago
> trick Normally fp numbers have an "invisible leading one bit", so an ieee fp is (1+0.nnnn) x 2^E where nnn is the mantissa and E is the exponent. as posits are always two's complement, you can model negative posits as having an "invisible leading two bit", aka (2-0.nnnn) x 2^E-1. this greatly simplifies circuitry for addition and multiplication. > the addition operation takes up quite a bit more floorspace, I miswrote, it's multiplication. This is simple: since the math requires floorspace of size n^2 where n is the maximum resolution of the mantissa, and posits have more resolution around unity. we did floorspace estimates for fp32 versus p32 and it looked like the extra space required for the mantissa did not overtake the savings from the crazy logic required for sign flipping, nans, and denormals that IEEE requires that posits don't. > I was hoping the current AI hype wave would bring some custom silicon for alternative float formats, Posits included i was hoping for this too!
- adgjlsfhk1 1y ago
  I really want a standards body to standardize a positized version of floating point numbers. Everything about the encoding of posits (i.e. NaR, no -0.0, exponent encoding) is so much nicer. The tapered precision part of posits is IMO the least interesting part.
  - dnautics 1y ago
    the twos complement thing is really elegant (-p == (posit -(int p))) in c notation
    - adgjlsfhk1 1y ago
      relatedly, having the same ordering as signed ints just makes everything so much easier. You can just use your integer radix sort to sort your floats without having to deal with sign and nans first!
      - immibis 1y ago

gsf_emergency_2 1y ago

2018 blog post on the FB effort: https://engineering.fb.com/2018/11/08/ai-research/floating-point-math/ https://engineering.fb.com/2018/11/08/ai-research/floating-p...