8 ms·
Beating Floating Point at Its Own Game: Posit Arithmetic [pdf]
- speps 9y agoI read the book on Type I Unums, it was quite interesting but I'm not an expert in this field. However, I appreciate the fact you don't have to buy a book to find his new research.
- CleanCut 9y agoThat looks really cool. Is there hardware support for posits anywhere?
- jamesaross 9y agoJohn Gustafson is on the board of Rex Computing, a small semi startup. They claim to have taped out last year, but I don't know if the chip has been validated or if it had posits in silicon, but I know that is one of their longer term goals. I find Gustafson's UNUMS 2.0 more compelling than the first version.
- nayuki 9y agoJohn's presentation more or less mirrors the topics covered in the paper: https://www.youtube.com/watch?v=aP0Y1uAA-2Y https://www.youtube.com/watch?v=aP0Y1uAA-2Y
- trsohmers 9y agoFounder of REX Computing here, we taped out in July of 2016 and got our silicon brought up and working back in February. I gave a talk at Stanford showing our hardware and a tiny bit of software: https://www.youtube.com/watch?v=ki6jVXZM2XU https://www.youtube.com/watch?v=ki6jVXZM2XU Type 2 unums are pretty much entirely deprecated by type 3 unums (now given the name 'posits' as referred to in the OP's linked paper)... they are basically superior in every way, and I recommend watching John's February 2nd talk at Stanford.
- deepnotderp 9y agoDoes the Rex Neo have posits?
- delhanty 9y agoHN discussion from 2 years ago: https://news.ycombinator.com/item?id=9943589 https://news.ycombinator.com/item?id=9943589
- deleted 9y ago[deleted]
- ktta 9y agoThis is different from unums which it says so in the second line of the Abstract.
- shpx 9y agoActual discussion of posits from 3 months ago https://news.ycombinator.com/item?id=14013971 https://news.ycombinator.com/item?id=14013971 John Gustafson (the author) has an account https://news.ycombinator.com/threads?id=jlgustafson https://news.ycombinator.com/threads?id=jlgustafson
- delhanty 9y agoApologies - my bad.
- deleted 9y ago[deleted]
- copperx 9y agoHere's a C/C++ partial implementation of posits: https://github.com/libcg/bfp https://github.com/libcg/bfp
- akkartik 9y agoThanks! Can you elaborate on what's missing?
- libcg 9y agoConversion from/to float and double, multiplication, division and reciprocation are implemented and should be working properly. The next big TODOs are addition, substraction and rounding. Then advanced operations like sqrt, fma, etc. As part of the posit spec. I've been working alone on that project, this being my first time working with floating point representation. I'm happy I could even get to that stage, and I hope someone can pick up my work and start contributing.
- dnautics 9y agobfp uses C++ classes which store the format hyperparameters as fields. If you'd rather a C/C++ library memory layout which reflects the bitwidth of the particular posit: https://github.com/Etaphase/FastSigmoids.jl https://github.com/Etaphase/FastSigmoids.jl contains both a julia library and a C/C++ library. Posits are implemented as a C type and a C++ class where the type and class sizes correspond to the bitwidth.
- libcg 9y agoDev here, I'm planning to transition to a C library with fixed size posit implementations. I need all the help I can get :) FastSigmoids uses float and double types which would be problematic in embedded environments.
- dnautics 9y agoShoot me a message. I am planning on making a bitwise/intmath implementation in C. Things might be accelerated if you know someone interested in sponsoring it.
- yorwba 9y agoThe LINPACK benchmark feels a bit like bragging. Of course you can demand an exact dot-product operation for any implementation of your standard, and then use it to compute an exact vector-matrix product, but floats could do the same if it were part of the standard. The fact that it requires a 1024-bit accumulator makes me doubt that it would be used for a massively parallel implementation e.g. in GPUs. The overall idea of choosing a number distribution that loses less precision for common operations and orders of magnitude is pretty compelling, though.
- dnautics 9y agoThe accumulator is pipelineable. For most graphics and ml applications you probably don't need the exact dot product. You probably do want it for scientific applications... There is a trade-off in performance, as always, an appeal to calculation speed though falls to the retort of, sure, you can calculate wrong answers with higher throughput if you wished.
- deleted 9y ago[deleted]
- Dylan16807 9y ago> There are no “NaN” (not-a-number) bit representations with posits; instead, the calculation is interrupted, and the interrupt handler can be set to report the error and its cause, or invoke a workaround and continue computing, but posits do not make the logical error of assigning a number to something that is, by definition, not a number. This simplifies the hardware considerably. What a strange claim. Outputting a NaN is easier than raising an interrupt, or could be done within an interrupt, and detecting a NaN input requires a handful of gates or less. This is not to say that NaNs are a good or bad method, but they're definitely not expensive to implement. > posits lack a separate ∞ and −∞ [...] “negative zero” is another defiance of mathematical logic that exists in IEEE floats. I will note that the IEEE standard almost had a projective infinity mode, and x87 has that mode. > floats are asymmetric and use those bit patterns for a vast and unused cornucopia of NaN values If we want to ignore languages like javascript and lua, sure.
- RhysU 9y ago> If a programmer finds the need for NaN values, it indicates the program is not yet finished... These guys have never allreduced a timestep in a simulation where something can go aphysical. I mean, sure, one can communicate aphysical conditions using out-of-band means. But it is so pretty when it just rides along for free as a NaN result. Edit: I stand corrected having looked up authors. I still think having float effectively be Maybe[float] is a feature not a bug.
- psykotic 9y agoQuiet NaN is critically different from the Maybe monad's Nothing value. Maybe gives you proper exception semantics even if it's implemented by means of predicated data flow rather than early-out control flow (and in a pure, non-strict language like Haskell there isn't a real distinction between data flow and control flow). Whereas quiet NaNs have a significant issue with non-termination where you have to very carefully express all your loop conditions positively. For example, 'while (x - y > eps)' will terminate if either x, y or eps are NaN, but any code that relies on 'if (positive_condition) break' and similar constructs for termination will loop or recurse forever if the condition operands become NaN. A closer analogue of NaN might be the x ("unknown") value in the three/four-valued logic of hardware simulation languages like Verilog. But the situation in Verilog, despite its challenges, is so much better since you don't have a hard discontinuity between "numbers" and "logical values". Numerical comparisons between unknowns can produce unknown logical values, and you have equations like true & unknown = unknown, true | unknown = true, ~unknown = unknown, etc, so you don't have the issue with NaNs where as soon as something crosses from the floating-point realm to the integer or logical realm, it has to commit to a definite value, with all the aforementioned issues. All that said, NaNs were the right design choice given the constraints. But it's far from ideal.
- stephencanon 9y agoPosits are actually quite reasonable, but there's a lot of either ignorance or disingenuousness in this article, which is really too bad. I wish that John would ditch the hyperbole and solicit feedback from other experts, because posits are not a bad idea, but the presentation continues to give him the trappings of a crank. I'll unpack just the first example that jumped out at me: > Currently, half-precision (16-bit) IEEE floats are often used for this purpose, but 8-bit posits have the potential to be 2−4× faster. An important function for neural network training is a sigmoid function, a function f(x) that is asymptotically 0 as x → −∞ and asymptotically 1 as x → ∞. A common sigmoid function is 1/(1 + e−x) which is expensive to compute, easily requiring over a hundred clock cycles because of the math library call to evaluate exp(x), and because of the divide. "have the potential to be" 2-4x faster? Sure. But until we see an implementation, 1-2x is much more likely (closer to the 1x end of the spectrum). Commodity hardware runs 32b IEEE float multiplications with 3-4 cycles latency and single-cycle throughput (or better). 16b can be made faster if designers care to. There's simply not much "faster" available for posits to inhabit (2-4x faster than 16b float would be faster than small integer arithmetic). Evaluating a sigmoid function requires "over a hundred clock cycles" only in the most naive possible implementation sitting on top of a lousy math library. Using 32b floats on a current generation phone or laptop with a decent math library, a naive scalar implementation of the sigmoid function has a latency of less than 50 cycles. But latency doesn't matter at all in a machine learning context; we're interested only in throughput. On a machine with AVX-512, a single core can evaluate a sigmoid function with a throughput of about 1.25 cycles / input. In full-precision 32b floating-point (i.e. a relative error of ~10^-7). John's proposed posit implementation has a relative error of about 10^-1. If we target that error threshold, we can trivially go below 1 cycle/input in 32b or 16b float on a phone. So IEEE floats are at least two orders of magnitude faster than he claims. You need to go back more than 15 years for the numbers that the paper tosses around to even be plausible. There are several other examples like this in the paper. I don't want to be too antagonistic, because posits are not a bad idea (actually, I think they're a pretty good format), but this paper is either ignorant of the state of the art, or more marketing than science.
- dnautics 9y agoI'm just going to be blunt here. John and I have decided that we need to be more marketing savvy after he's had trouble with several rounds pitching other floating point formats. Posits are just an intermediate step to try to build acceptance for valids, so there's a lot of effort put into branding. A couple of points: 2-4x faster means 2x faster in dot product based simd and 4x faster in matrix simd, assuming that your bottleneck is memory throughput. The sigmoid function realistically isn't a bottleneck in general, but you gotta admit it is pretty cool to have a ~zero clock cycle approximation. (I've tried to rein in John a bit on this one)
- strigeus 9y agoWhat's the smallest positive integer that 64-bit posits cannot represent exactly?