14 ms·
Java’s floating-point hurts everyone everywhere (1998) [pdf]
- PaulHoule 3y agoFor general purpose computing I consider floats with base-2 exponents harmful and that's worse than the little things Java does wrong.
- db48x 3y agoIEEE 754 defines decimal floats in addition to binary, but you’ve never used them because no x86 processor has provided hardware implementations of them.
- yoz 3y agoAnd I always thought that IEEE 754 was the problem. Thank you for opening my eyes!
- FabHK 3y agoFor general purpose computing (approximating real quantities) floats with base-2 exponents are just right on base-2 computers. For dealing with money (and other quantities that are fundamentally integer), they might be the wrong choice, but that's well understood and sort of obvious (use integers to handle integers).
- PaulHoule 3y agoI disagree. There are many non-professional programmers who get baffled by various "computerisms" such as 0.1 + 0.2 != 0.3 Excel tries to hide this but ends up doing even stranger things if you push it hard enough. I think a lot of people who could use computers to put their skills on wheels just give up because of this "lack of empathy" that manifests here and in other places. If you do 0.1 + 0.2 - 0.3 on a pocket calculator you get the right answer and you should get the same right answer in a Jupyter notebook. The only person who should be exposed to the base 2 arithmetic of the computer is a professional programmer who knows assembly language. There are numerous social consequences of this that are harmful such as the perception that computer programmers are "grinds" and "nerds" and the idea that "idea people" are more worthy than the people that execute, etc.
- kragen 3y agooh, i thought you meant base 2 as opposed to base 16 (like the 360) there is a significant performance penalty for bcd arithmetic, so bcd floating point has never been attractive to the customers for floating point: cray buyers, fortran programmers, gamers, analog circuit designers, climatologists those people don't really care about the beginner issues you mention if you're willing to accept a performance penalty in exchange for easier learning, you can implement decimal arithmetic in software; many people have, and it's in the cobol and sql standards and the python standard library. the version i implemented in bubbleos is 83 lines of c, https://gitlab.com/kragen/bubbleos/blob/master/yeso/decimal.h https://gitlab.com/kragen/bubbleos/blob/master/yeso/decimal.... https://gitlab.com/kragen/bubbleos/blob/master/yeso/decimal.c https://gitlab.com/kragen/bubbleos/blob/master/yeso/decimal.... and compiles to about 1.3k of code but a lot of us are using cpython in jupyter to prototype algorithms we want to run as fast as possible, so we want it to behave like the floating-point hardware does
- lifthrasiir 3y agoYou are entirely right that user-facing programs should strive to hide IEEE 754 artifacts (see, for example, the Android calculator app that uses exact real arithmetic). But if you are programming, there are tons of quote-unquote computerisms that we have for a good reason but are not really intuitive for newcomers. The whole concept of variables, pointers and general indirections, zero-based indexing, (pseudo)randomness, Unicode, time complexity, concurrency and parallelism and so on. Many (but not all, I admit) professional programmers take them as granted but they are just as arcane as the concept of base-2 floating points.
- moonchild 3y agoIf somebody does not understand the implications of a computation such as a floating-point 0.1 + 0.2, they should not be using floating-point. It's not the job of the language to protect them from that. Decimal floating-point will be prone to very similar malfeasances, if that is what you are proposing; somebody unaware of the issues of binary floating point will not be helped by decimal floating point. I do think it would be a good idea for programming languages to expose more exact number representations (for example, rational numbers, which are supported by some programming languages). But numbers are complicated, and you will never be able to imbue a computer with a number type that always behaves exactly the way that someone who knows nothing of computers would naively expect. Even mathematica does not quite manage it, and mathematica takes some somewhat extreme measures which would not be considered acceptable in many other general-purpose languages.
- constantcrying 3y agoThe floating point standard is really good at pretending to be real numbers on a computer. Every problem is the mathematical consequence of its features.
- pestatije 3y ago1998
- dang 3y agoAdded. Thanks!
- verteu 3y agoI recommend reading from Page 19, it describes the heart of the issue.
- deleted 3y ago[deleted]
- breadwinner 3y agoThis PDF is dated 2004. Please add that to the title.
- shagie 3y agoFrom the last page: > Conclusions > ... > To win, Java has to surpass Microsoft's J++ in in attractiveness to software developers. This means better design better thought through, less prone to error, easier to debug, ... and many other things. This document is a bit dated.
- kazinator 3y agoIf it talks about J++, it's ancient. You helped people dodge a time-wasting bullet here.
- 6510 3y agoFloats are such an embarrassment to have to explain to new developers. Then you have to explain that there is no intention to fix it. And then there are all these developers who make the field look like a cult by endlessly chanting how it works the way it should work. Who seem not to know the difference between correct and incorrect, the difference between right and wrong.
- FabHK 3y agoThere are infinitely many real numbers (uncountably even), but your computer can only handle finitely many things. Something has to give. While many people complain about it, IEEE 754 is a very thoughtful and thorough standard, and the reason it is still around is that nothing unambiguously better has been proposed yet.
- Groxx 3y agoFor the vast majority of uses: big-number libraries are far better. They act like people expect numbers to act. Save floats for performance optimizations. Start out correct, optimize when you can show both need for speed and non-need for correctness.
- galangalalgol 3y agoIf you are doing signal processing, ML, simulations, gaming or the like, you can know going in that you will need to use primitive types from a performance standpoint which sometimes means fixed point but often means floating point.
- aidenn0 3y agoMany simulations and gaming are both better off using fixed point. The range available with a 64-bit integer is staggering and precision varying with distance from an arbitrary origin is a bug, not a feature for those cases.
- 3y ago
- marcosdumay 3y agoIsn't that just inherited from C? I can't think on any widely-used language that does it right nowadays (except Rust, if you change some flags, and not by default). For sure, 20 years ago there were some dying languages that behaved differently, and yes, the Java position there did hurt everybody, but it's the same one everybody else took.
- galangalalgol 3y agoWhich flags?
- lifthrasiir 3y agoYou must be talking about integer overflow checks [1]. This talk is more about something like C's fenv.h. [1] https://doc.rust-lang.org/rustc/codegen-options/index.html#overflow-checks https://doc.rust-lang.org/rustc/codegen-options/index.html#o...
- jcranmer 3y agoSo half the complaint here is that Java doesn't let you use the 80-bit x87 floating-point type. Well, actually it does--if you don't add `strictfp` to your method declarations, Java is allowed to compute intermediate results of float and double computations in higher precision, and this was retrofitted because it's actually somewhat difficult to get the x87 floating-point unit to actually do single- or double-precision (as opposed to extended-precision) arithmetic. But you don't actually want to use the 80-bit type; it's slower in practice, and it has some extra wonkiness over other IEEE 754 types that it's really not worth it. And on 64-bit x86, everyone just uses SSE for floating-point units and thus you don't need to worry about x87 unless you explicitly opt into its insanity. As a numerical analyst, Kahan is pretty obsessed with use-as-much-precision-as-you-can. But there's a useful rule of thumb: you need about twice the amount of working precision as your final result. Since double precision has a 53-bit mantissa (~16 decimal digits), that means if you need only 8 or fewer decimal digits, you're completely fine with double precision. And furthermore, the experiences I've had with many programmers suggest that getting bit-equivalent results from different machines is a higher priority than squeezing the best possible numerics out of your hardware. HPC does tend to care about the latter a lot more, but that's also an area where the solution is almost always to just use your system's advanced math libraries (e.g., MKL for dense linear algebra).
- thebooktocome 3y agoStrictfp is no longer used as of JVM 17. IEEE-754 semantics are default. https://openjdk.org/jeps/306 https://openjdk.org/jeps/306 Ironically, using your system’s math libraries will probably make replicating floating point results harder. Especially on macOS. Accelerate is a curse. I disagree with your gloss of Kahan’s philosophy. His approach is more along the lines of “do not waste precision”. But this philosophy is not the complete truth; as close as I can state it briefly, my modification would be “do not waste precision that may be needed later”.
- jcranmer 3y agoI was unaware that Java finally got rid of strictfp (though happy that it finally did). It was added in Java 1.2, perhaps in response to this paper, though I don't know the entire timeline accurately.
- lifthrasiir 3y agoThis is from March 1998, and predates JVM 1.2 which introduced strictfp (which is important because JVM was otherwise numerically IEEE 754 compliant before strictfp). I believe the core point of this talk has not been "fixed" [1] because it completely missed the problem with IEEE 754 global states. Like, the talk says: A flag is a type of global variable raised as a side-effect of exceptional floating-point operations. Also it can be sensed, saved, restored and lowered by a program. When raised it may, in some systems, serve an extra-linguistic diagnostic function by pointing to the first or last operation that raised it. Any modern programmer knows that side effects are, while inevitable, hard to tame and some discipline is needed. Pure functional programming, mutable XOR shared, software transactional memory, you name it. This part of talk completely handwaves a difficulty of side effects and forces every language to be handcuffed with those global states and side effects. No good. [1] Joe Darcy, one of the students of William Kahan, later went to Sun to improve fp support in Java; see https://web.archive.org/web/20090402234711/http://blogs.sun.com/darcy/entry/everything_old_is_new_again https://web.archive.org/web/20090402234711/http://blogs.sun.... and https://web.archive.org/web/20100224062224/http://blogs.sun.com/darcy/entry/everything_older_is_newer_once https://web.archive.org/web/20100224062224/http://blogs.sun.... for example. But his work was more about library supports AFAIK.
- smarks 3y agoThanks for mentioning Joe Darcy. He continues to work on the JDK at Oracle and occasionally works on numerics issues. Notable for this discussion, he recently updated the Java specifications for JDK 17 to remove non-strict FP semantics (which also entailed removal of the `strictfp` modifier, since the default now is always strict). See JEP 306. [1] [1]: https://openjdk.org/jeps/306 https://openjdk.org/jeps/306
- segfaultbuserr 3y agoThe problems were not specific to Java, they also exist in most modern programming environments, including FORTRAN and ANSI C. Java just happened to be the hottest language in the news in the late 90s, so the author decided to specifically target Java - "hurts everyone everywhere" was a wordplay of Java's advertisement. The main author of this report, W. Kahan, was also the original author of IEEE-754. He was (and is) strongly unsatisfied on the state of floating-point in practical systems, and it was not his first time to complain about these problems. One can find his most recent critique from a few years ago. My understanding is that, Kahan's IEEE-754 is meant to: 1. The use of double and extended precision should be encouraged to safeguard non-experts from floating-point errors. Everything should be at least double precision by default, and extended precision serves as an additional safeguard. In fact, when IEEE-754 was being drafted, Kahan believed 128-bit floating point should also be supported as computers become more powerful in the future. The computational cost was too high at that time, so he settled on 80-bit as seen on the 8087. He criticized Java for not supporting it. 2. Floating-point exceptions should be used and turned on everywhere to safeguard programmers from making mistakes, and possibly to allow programs to handle them as special cases in the logic at runtime. He criticized Java for not supporting it. 3. Unsafe optimizations should not be done, such as automatically using FMA, or using algebraic identity in compiler optimization. He criticized Java for allowing it in some cases. Unfortunately, as far as I can see, these ideals of IEEE-754 have all but largely disappeared in real-world applications since then, for various practical reasons. The industry did not move to 128-bit floating point because its performance overhead is too large, the original assessment of IEEE-754 in the 1980s was too optimistic. Similarly, the industry did not accept 80-bit extended precision as a standard but saw it as an oddity of the Intel 8087, even Intel has abandoned it - x86_64's SSE or AVX has removed all support of that, making it an exclusive feature of obsolete i386/i686 machines. My impression is that anything above double precision is no longer used in the industry (IBM POWER does have native quad-precision support, to their credit). A secondary overhead of higher-precision floats is the memory wall, arguably a more serious problem today. Memory bandwidth has become the most serious overhead for many numerical programs. Modern computers have a machine balance of 100:1, it means you need to do as many as 100 floating-point operations after loading a single value from RAM to reach the machine's peak performance. But this is not compatible with many algorithms with an inherently low arithmetic intensity, including important physics simulations. The use of of FP80 or FP128 will make them unacceptably slower. As a result, today's trend is moving from FP64 to FP32, and even to FP16 or a custom 16-bit format if possible, not vice versa. The use of floating-point exceptions similarly became unpopular because of performance problems. I'm not an expert on this, but it was my impression that signaling a floating-point exception was so expensive on both hardware and operation-system level that it was never seriously used in most practical programs. So in contrary to IEEE-754, these exceptions never became an integral part of the programming environment. In addition to an expensive Unix signal, potential exceptions also inhibit the efficient pipelining of modern out-of-order, superscalar CPUs. Finally, the use of unsafe optimizations is prevalent in many applications when rigor is sacrificed for speed. So overall, the original spirit of IEEE-754 was long gone - for better or worse, unfortunately.
- dang 3y agoRelated: How Java’s Floating-Point Hurts Everyone Everywhere (1998) [pdf] - https://news.ycombinator.com/item?id=6585828 https://news.ycombinator.com/item?id=6585828 - Oct 2013 (72 comments)
- kstenerud 3y agoWow, that document is difficult to read. I was expecting a summary of the main points a the top, followed by a set of sections - each with cogent introductions and a progressive series of arguments to prove the point, followed by a concise conclusion. Instead, this is a rambling, mostly unstructured document where it's nearly impossible to follow the many scattered threads of thought, or even catch which side of some of the arguments he's actually on.
- bradleyjg 3y agoToday BigDecimal (arbitrary precision) should be the easy to use default and float the annoying to invoke exception. But it’s understandable why the designers wouldn’t have done that in the 90s.
- xxs 3y agoBigDecimal is the exception not the norm, and it's a lot slower than 'double'
- bradleyjg 3y agoMost of the time it doesn’t matter. Double (or float) by default is premature optimization. When you start dealing with millions or billions of values, sure reach for the annoying, counterintuitive, footgun laden numeric types. But that’s not most code.
- gpderetta 3y agothe sad state of software today shows that 'premature optimizations' aren't.
- bradleyjg 3y agoWould you prefer a world where software is orders of magnitude more expensive and so limited to a few applications like aircraft and central banking? Because that’s the alternative to your “sad state.”
- xxs 3y agoWell (tens) thousands doubles vs BigDecimals (think of marketing data) and any operations over them is a massive difference. The premature optimization is beyond uncalled for. Learning to use floating point type is something that most developers should do. No need for weasel words, either (annoying, counterintuitive).
- 3y ago
- exabrial 3y agoWhy doesn't Java (or other languages) support a fixed length decimal by default? Granted Java has BigDecimal, but why isn't there a basic type for this? For business and accounting systems, this seems like an obvious choice.
- manicennui 3y agoI'm always baffled by how many software engineers with CS degrees don't understand IEEE 754 floats. I had a coworker who was bugging the shit out of me claiming that Macs had a "bug" in several meetings because he didn't understand floats. Recently I heard someone else make the same claim about Ruby. WTF are they teaching people in CS programs?
- danhau 3y agoThey‘re teaching them that floats and doubles are numbers with decimal places, and that‘s where most stop. There‘s a reason articles like „Myths programmers believe about floating point“ are written and usually get highly upvoted.