6 ms·
NumPy QuadDType: Quadruple Precision for Everyone
- OutOfHere 2y agoIs there something here for people to use? Can the comparison also be done using the max precision unsigned integers instead of just float64?
- gh02t 2y agoLooks like it's not ready for serious use yet but the initial implementation is here https://github.com/numpy/numpy-user-dtypes/tree/main/quaddtype https://github.com/numpy/numpy-user-dtypes/tree/main/quaddty... . I was just complaining about long double on Windows working on an algorithm that can benefit from higher precision, this post is serendipitous.
- slashdave 2y agoIn my experience, it can be helpful for quickly fixing precision issues. Sometimes you just want to get on with your calculations and not go back and reformulate your numerical weak points. It's less necessary with good practice.
- jabl 2y agoAs an aside, the GNU Fortran compiler (gfortran) also provides a quad precision floating point type, similar to this one. If the target supports native quad precision it uses that, otherwise it falls back to software emulation. Math functions are provided via libquadmath. Newer glibc also provides quad precision functions with slightly different names, IIRC using a f128 suffix rather than "q" like libquadmath.
- Conscat 2y agoDoes anyone have examples of calculations that need quadruple precision? My last work place seemed fine using double precision for simulating satellites in orbit, and I have the imprecision double precision makes black holes and quantum physics simulation work correctly, so I have no intuition for this.
- gh02t 2y agoI see it in statistics. It's not uncommon in a lot of statistical algorithms to need to take the product (or sum) of a lot of very small numbers and then divide by that product when computing likelihoods of a bunch of independent measurements. There are some algebra tricks you can use to get around loss of precision like working in the log domain or some tedious scaling tricks, but working in quad precision can help when those aren't enough. The article also gives an example from quantum physics. Some chaotic or highly numerically unstable problems just need quad precision to not blow up; primarily problems/algorithms (can be the problem itself or the algorithm to solve it that causes the issues) that tend to catastrophically amplify small errors. Matrix inversion is another example, to the point that most algorithms try to do almost anything to avoid explicitly inverting a matrix. But sometimes you just have to.
- slashdave 2y agoMatrix inversion precision can be fixed iteratively (at a CPU cost, of course). Nearly singular matrices should probably be factored in any case.
- gh02t 2y agoProbably, that falls into "doing almost anything to avoid inverting a matrix explicitly", but direct inversion is the classic example here.
- kolbe 2y agoFor numerical PDE solvers, you never do the actual inversion, but the discretization of the system (denoted A) can have significant sensitivity to numerical precision. Look up Condition Number and Preconditioning. In fact, resolving the problems with numerical precision for some A takes as much time as actually solving the system, and it's worth it.
- awoimbee 2y ago> The long double type varies dramatically across platforms: > [...] > What sets numpy_quaddtype apart is its dual-backend approach: > Long Double: This backend uses the native long double type, which can offer up to 80-bit precision on some systems allowing backwads compatibility with np.longdouble. Why introduce a new alias that might be 128 bits but also 80 ? IMO the world should focus on well defined types (f8, f16, f32, f64, f80, f128), then maybe add aliases.
- 0cf8612b2e1e 2y agoReally seems like propagating the current platform inconsistencies into the future. Stick with 128 always, performance be damned. Slow code is much preferable to subtlety broken because you switched the host OS.
- gnulinux 2y agoEspecially if you need 128-bit float precision. It's very well known and understood that quad float is much slower in most platforms, extremely slow in some. If you're using quad float, it's because you absolutely need need all 128 bits, so why even reduce it to 80 bits for "performance"? Seems like an irrelevant design choice. Programmer can choose between f128 vs f80 if performance is intractable in the target platform.
- bee_rider 2y agoMaybe it depends on the application? If you have written a linear system solver, you might prefer to express yourself in terms of single/double precision. The user is responsible for knowing if their matrix can be represented in single precision (whatever that means, it is their matrix and their hardware after all), how well conditioned it is, all that stuff. You might rather care that you are working in single precision, and that there exists a double precision (with, it is assumed, hardware support, because GPUs can’t hurt us if we pretend they don’t exist) to do iterative refinement in if you need to.
- JanisErdmanis 2y agoUnlikely to be user here, just curious. If one would be willing to use numpy_quaddtype with scipy or other libraries that depend on the numpy would that be possible without adding this library as dependency to them?
- kolbe 2y agoUntil my hardware implements f128, even if I had a solid use for it, I cannot sacrifice the speed relative to f64.
- sfpotter 2y agoThe only real place for quad precision is when you're working with numbers that truly require you to be able span a massive difference in magnitudes without round-off error accumulating. For some context, an electron "has structure" roughly on a length scale of about 10^-15 meters (details here unimportant). If you're representing a quantity in O(1) meters, then doing a handful of floating-point operations in double will likely accumulate error on the order of O(1) electrons. Incredible accuracy! I think the key here is to be mindful of units and make sure you're combining like with like and scaling things appropriately as you go. If you're doing things the wrong way, quad will punish you eventually, too. An underappreciated tool for working more confidently with floats is interval arithmetic. Worth checking out if you actually (truly) need to do things robustly (most people don't).
- andrepd 2y agoThat's one of the (many) problems with ieee floats: they "waste" as much bits for mantissa precision in the range of 10^0, 10^1, 10^2, etc, as they do in 10^300 or 10^-300! Proposals like Posits aim to improve on this :)
- nestorD 2y agoIn practice, Posits grabs very few additional bits and those are mostly felt in low bits (16 and less) regimes. That's a big reason it is mostly seeing adoption in memory constraint regimes. A great thing in the Posit proposal is that they also consider a distinct accumulator type for things like sums and dot product. In my experience (I did a PhD on measuring floating point error in large numerical simulations) those operations are the ones that are most likely to sneak on you and need fixing. But most people don't realize that those accumulator type are actually easy to use with IEEE floats, they are entirely orthogonal to using Posits.
- sfpotter 2y agoIEEE floats are an incredible feat of engineering. The people who designed them were geniuses. We should be appreciative of the fact that we have a consistent and portable way to do numerics. I'm glad we've got IEEE 754 and don't have to suffer through the pain of numerous competing standards.
- jofer 2y agoI am very happy to see that the article _immediately_ explains how this is different from long-existing np.float128/np.longdouble. I was scratching my head for a bit there... It's also good to see that custom dtypes have come a long way. This is a great application of a custom dtype, and that wouldn't have been possible a few years back (well, okay, a "few" to me is probably like 8 years, but still).
- Archit3ch 2y agoShould be fun for audio circuit simulations!
- nestorD 2y agoI have done my PhD on measuring floating-point error in large numerical simulations. In my experience, most of the time, if 64-bit floating points are not enough, then trying to increase precision further is unlikely to help much (there are exceptions). In general, using a well-placed compensated (or exact) sum/dot product algorithm will solve most of your problems at a low cost. Nowadays, most people are familiar with Kahan summation (and related tricks like sorting numbers), but they do not know that it does not help much (often less than 128-bit floating points) and that the state of the art for those algorithms has moved quite a bit since. Nowadays, you can ensure that your sum/BLAS operation produces a result within one ulp of the exact result! I highly recommend perusing the Handbook of Floating-Point Arithmetic (the Accurate Algorithm online page also seems fine [0]) to get an idea of what can be done. Implementations of these ideas can be found in most languages used for numerical computation (a quick Google search gave me options for C++/Python [1], Rust [2], etc.). [0]: https://accurate-algorithms.readthedocs.io/en/latest/index.html https://accurate-algorithms.readthedocs.io/en/latest/index.h... [1]: https://github.com/yafshar/xsum https://github.com/yafshar/xsum [2]: https://github.com/bsteinb/accurate https://github.com/bsteinb/accurate
- PaulHoule 2y agoThat’s what I was taught about numerics doing my PhD in the 1990s. I was taught that you shouldn’t even use doubles if you could use singles.
- sebzim4500 2y agoThat's also my experience. It's interesting that 64 bits are often necessary (as opposed to 32) but 128 bits so rarely are.
- nestorD 2y agoYes! I suspect that it boils down to 32 bits not being that much (you can enumerate them to test a function), while 64 bits is a lot. By that point, if you are having problems, then you are likely hitting a wall that will not be crossed by throwing additional bits at the problem.
- 2y ago
- Archit3ch 2y ago> The SLEEF (SIMD Library for Evaluating Elementary Functions) backend > Significand: 113 bits > Exponent: 15 bits Seems like a worse tradeoff than Dec128 [1] (110 bits, 17 bits) unless it's way faster. [1] https://www.intel.com/content/www/us/en/developer/articles/tool/intel-decimal-floating-point-math-library.html https://www.intel.com/content/www/us/en/developer/articles/t...