12 ms·

Do Developers Understand IEEE Floating Point? [pdf]

116 points by matt_d 8y ago

jdblair 8y ago
Obligatory link to the classic paper on the subject, from 1991: https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.h...
gcc_programmer 8y ago
60% of the people who took the survey develop software to support their main role. So, 60% of the people are not really software engineers but coders by necessity: could this explain the results easily?
kbumsik 8y ago
Although understanding floating point is important, it is often ignored in Data Science schools. The title reminds me a friend of mine who attends a data science master course without computer background. He asked me for help for his numpy homework on Bayesian Probability theory. The number he got was so small that he got weird result. I immedieatly thought it was because float64 couldn't cover that number so I told him to use longdouble dtype (it's 80 bits in Intel I think) and the problem solved. Later he told me even the professor didn't expect such case when announcing the homework and many students couldn't finish the hw on time because of this.
- de_watcher 8y ago
  That's a difficulty that exists for the simulations, optimization problems and various algorithms. Some need so many bits that they don't work at all.
- gronne 8y ago
  No such thing as “Data science” - just a fancy term for trade school computer skills. Thats why.
  - wolfgke 8y ago
    > No such thing as “Data science” - just a fancy term for trade school computer skills. Thats why. Relevant xkcd: https://xkcd.com/435/ https://xkcd.com/435/ Or as my friend put it: everything is just a trivial special case of either mathematics or physics. ;-)
    - gronne 8y ago
      True :) I know the strip. tbh I just felt like trolling the insecure “data hackers” crowd. Looking at downvotes: I think it worked.
    - everybodyknows 8y ago
      Fun xkcd, but mathematics does not belong on a continuum of the scientific disciplines, all of which seek to describe nature. "Die ganzen Zahlen hat der liebe Gott gemacht, alles andere ist Menschenwerk" https://en.m.wikipedia.org/wiki/Leopold_Kronecker https://en.m.wikipedia.org/wiki/Leopold_Kronecker Verbose, highly literal translation: "Loving God created the absolute integers; all else is the work of humanity."
- mehrdadn 8y ago
  Would the log-sum-exp trick have helped by any chance? https://en.wikipedia.org/wiki/LogSumExp https://en.wikipedia.org/wiki/LogSumExp
  - kbumsik 8y ago
    I didn't know this (I'm not in ML/Data Science) but that looks very helpful. I looked through his learning materials but I have never seen this one though.
    - mehrdadn 8y ago
      Yeah I only happened to know because of ancourse I took that taught some ML. I mentioned it because, later on, I even came across a situation where it would've been directly applicable, but for some reason the thought that it might solve the problem escaped me at that moment, so I asked and someone else had to remind me of that fact... so I feel it's probably not the easiest thing to recognize when it's happening.
- Veedrac 8y ago
  > I immedieatly thought it was because float64 couldn't cover that number so I told him to use longdouble dtype (it's 80 bits in Intel I think) and the problem solved. This is a very dangerous habit. If 64 bits isn't enough you are generally using the wrong representation.
Upvoter33 8y ago
60% of the time, developers understand floating point 100% of the time.
caseymarquis 8y ago
I once encountered an old database using floats as its primary keys. This was in a fairly popular piece of manufacturing software from the 90s.
- shabbyrobe 8y ago
  Wow, I'd love to hear more about this.
- mcv 8y ago
  Useful in case you want multiple entries with an id of approximately 1?
  - Const-me 8y ago
    Potentially, might be useful if you want small fixed-length keys that are fast to compare, and also you want to be able to insert new items between existing ones.
    - jmiserez 8y ago
      There was this article posted a while ago that used a Stern-Brocot tree and/or floats for ordering: https://news.ycombinator.com/item?id=16635440 https://news.ycombinator.com/item?id=16635440 It doesn't work in the pathological case though as any float has a fixed number of bits, and I see the article has been amended with a note.
- maxxxxx 8y ago
  What type of data would that be? I once did this when I did my own primitive version of a geospatial database in the 90s.
- sevensor 8y ago
  I sometimes, in Python, cautiously use floating point numbers as dictionary keys. I do this when what I mean is "this exact floating-point number" and not "the result of an equivalent computation." It's almost never a good idea, because it's really easy to trip yourself up by assuming the latter when only the former actually holds.
jstewartmobile 8y ago
Even if you understand them... in a polyglot world, you're still going to get burned. It is so easy--especially when you're jumping from a NUMBER language like SQL or Scheme to a floating-point one like C or JavaScript--to backslide into boners like for(i = 0.0; i < 1.0; i += 0.1) which would be a non-boner with a more advanced number tower. edit: My first Clojure program was the last time this got me--LISP on the outside, Java floats/ints/bigints on the inside.
- heavenlyblue 8y ago
  Yeah, and that’s why we have int for counting.
  - jstewartmobile 8y ago
    So if I'm marking an axis from 0 to 0.9 in 0.1 increments, I should obviously do the integer indirection dance? After all, every damn fool knows--even ones like myself who've implemented FP in assembly--that 0.1 + 0.1 + 0.1 = 0.30000000000000004. Silly me!
    - heavenlyblue 8y ago
      >> So if I'm marking an axis from 0 to 0.9 in 0.1 increments, I should obviously do the integer indirection dance? Yes. Because you clearly don't understand what you're doing at all if you believe that equivalence relation is defined the same way for floating-points as it is defined for integers. Floating-points aren't numbers. They're more like confidence intervals - and at a certain limit a confidence interval behaves just like an integer. But it's only a limit. The fact that most users of programming languages don't want to keep track of that in their code only reinforces my belief that they must do exactly as the software guidelines tell them to.
      - jstewartmobile 8y ago
        I know what floating point is. I've implemented it in hardware. I've implemented it in software. But you know what? Floating point is not the only way people have ever represented real numbers on the computer. In most SQLs, decimal fixednum types are predominant. Most LISPs have a number tower that automatically juggles fixnums, bignums, ratios, floats, and complex numbers. In the 8/16-bit days--before everything had a floating-point unit--it was not uncommon to store and calculate numbers directly in BCD. Two-digits to the byte, and all of the operations--including formatting--were very straightforward to implement. Many of the old business mainframes even have hardware instructions to pack/unpack/operate on BCD. > that most users of programming languages don't want to keep track of that... Why would I want to keep track of that when the language (or more accurately, the data type) will do it for me? I'm sorry man. The whole world isn't C and JavaScript (yet). That we have forced crud-app developers to resort to garbage like this[0]--in places where the performance and storage benefits of floating-point are irrelevant--is a stain on the profession. [0] http://www.jacklmoore.com/notes/rounding-in-javascript/ http://www.jacklmoore.com/notes/rounding-in-javascript/
mysterydip 8y ago
If you're like me and struggled to understand the traditional textbook floating point formula explanation, Fabien Sanglard (of doom/wolf3d black book fame) wrote a page that finally made sense for me: http://fabiensanglard.net/floating_point_visually_explained/index.php http://fabiensanglard.net/floating_point_visually_explained/...
- Waterluvian 8y ago
  Can I ask an embarassing question to verify I'm not misunderstanding that page? Does this mean that floats are less precise the larger they are? You have as many bits to describe the "position of the number" between 0-1 as you do between 2048-4096
  - Dylan16807 8y ago
    > You have as many bits to describe the "position of the number" between 0-1 as you do between 2048-4096 You mean 0.5-1 0-1 is about a quarter of the numbers, just as many as 1-inf
  - pfortuny 8y ago
    They are less precise in the absolute sense (the absolute error is larger) but equally so in the relative sense (the relative error is the same). Well, essentially, of course.
  - Sharlin 8y ago
    This is precisely what the term floating point means. The decimal point "floats" right or left depending on magnitude due to the mantissa-exponent representation. The opposite is fixed point where you have a fixed number of bits reserved for the whole and fractional parts. A 32-bit floating point can represent a vastly greater range of magnitudes than a 32-bit fixed point, with a vastly greater precision for values near zero, but at the expense of absolute precision at large magnitudes.
    - kps 8y ago
      At the computer floating point was developed, people understood it implicitly, because it was just an automated version of their current tools: log tables and slide rules. And likewise they knew better than to use floating-point where it doesn't work.
jmiserez 8y ago
Not developers. This survey included mostly PhD students (73), faculty (49), but only 23 software engineers. The rest are others. Only 1/4 see software engineering as their main role according to the paper. A PhD student is very different from a professional developer with years of real-world experience in industry. It's a completely different job. The title is highly misleading.
- crispyambulance 8y ago
  It's OK, not misleading at all, and the abstract spells out what they're studying and the motivation behind it clearly. A "developer" is just someone who writes software and not necessarily a "professional developer" with years of experience in a particular kind of job like what you have in mind. Moreover, I would even expect that the vast majority of "professional software engineers" who make a living strictly writing software without any particular domain in focus, might actually fare worse in floating point gotchas than scientific domain workers who are "software engineering dilettantes" but nonetheless do computation as their job.
  - deleted 8y ago
    [deleted]
  - jmiserez 8y ago
    Maybe. The paper even mentions this, sort of contradicting your expectation: >Perhaps not surprisingly, the most predictive factor is simply Contributed Codebase Size, the effect of which is shown in Figure 16. >There is a gain of only about 2/15 compared to those who reported a codebase where floating point was not intrinsic or where they were not involved. The title would be better suited if they included a higher number of HPC/ML/etc software engineers (post PhD) that work on larger scale projects in industry/labs. Specifically, I disagree with this assessment: >We believe that the combination of our recruitment process and the resulting background of the participants illustrated here suggest that our sample is a good representative of software developers who write code for, and in support of, science and engineering applications. That's just outright ignoring all science and engineering applications outside of academia/academic research.
abductee_hg 8y ago
according to this document the people who implemented java did not: "How Java’s Floating-Point Hurts Everyone Everywhere" https://people.eecs.berkeley.edu/~wkahan/JAVAhurt.pdf https://people.eecs.berkeley.edu/~wkahan/JAVAhurt.pdf p.s.: it's by William Kahan, who the primary architect behind the IEEE 754-1985 standard for floating-point computation (and its radix-independent follow-on, IEEE 854)
- nraynaud 8y ago
  no language has a good support for IEEE754. The only doable thing is using some weird macros in C/C++, but the rest of the languages ignore the issue altogether. And LLVM too.
  - ChrisLomont 8y ago
    C/C++ is not required to use ieee754 as it's float format, so your code would always run under that cloud. Fortran, D, Factor, and SBCL likely have better support. But anything where a compiler can reorder things is suspect, if you need exact behavior, since even simple register spilling causes trouble (Intel), among other things compilers may do.
    - Asooka 8y ago
      AFAIK C and C++ compilers are not allowed to re-order floating-point arithmetic precisely for that reason. Also, given that all cpus support SSE/Neon/etc. now and I haven't seen a compiler use the 387 FPU, I suspect register spilling shouldn't be an issue these days. Of course, legacy/exotic platforms will deviate from the norm in fun ways.
      - ChrisLomont 8y ago
        >AFAIK C and C++ compilers are not allowed to re-order floating-point arithmetic precisely for that reason. You don't have to reorder to make bugs. Simply having a compiler replace (a-b)+b with a is a bug. Simply flushing an internal 80-bit floating register to a 64 bit memory place, then loading it, causes differences in computations. Note the C++ spec specifically states it does not require an underlying representation or even any certain level of accuracy (6.7.1.8). So many of these bugs are allowed. The spec does not even require that compile time evaluation of an expression must match a runtime evaluation of the same expression (8.20.6). When you hit that in code you're going to be surprised. C/C++ is not required to round-trip floats/doubles when printed and parsed. stream formatting rounding modes are allowed (and are) not exact, leading to different behavior on formatting the same number on different systems. Dealing with these led to compiler implementation defined behavior added due to market pressures, making C/C++ numerics less of a mess, not anything in the spec. C/C++ has notoriously been bad at all this, which is why the history of it is littered with such bugs, errors, undefined behaviors, and dozens of compiler switches to try and mitigate such behavior. For example, many compilers notoriously replaced high-precision Kahan summation with low-precision regular summation because optimizations tell the compiler that such things are allowed. The state of the art in C/C++ is now passable, but it is still flawed, and has been historically very bad. And all this met the C/C++ spec of the time. Because if the underlying representation does not require IEEE 754, then it cannot require only transformations for IEEE 754 without making other representations behave badly.
known 8y ago
Yes with http://0.30000000000000004.com/ http://0.30000000000000004.com/
Traster 8y ago
This is a pet subject of mine since as an FPGA developer a huge amount of my time is spent focusing on bit-precision and Fixed-Point and reduced precision floating point implementations of algorithms. In fact one of the interview questions I ask focuses on how to do different mathematical operations in floating point using only logical/integer operations. On the one hand I find the results unsurprising - even people I know to have worked a lot on numerics often have only a rudimental understanding of the intricacies of corner case behaviour for floating point, and yes that absolutely manifests in hitting a wall when something curious goes wrong. Mostly this results in head banging until you find the piece of code going wrong is the numerical piece and then very quickly you start looking at possible floating point gremlins. Having said that though this paper seems to have a very academic view of what HPC is. Even for people designing HPC systems numerical optimizations are rarely a huge chunk of their job so it's probably not actually that important - I think the fact few people have a good understanding is a reflection of the fact it's not often necessary. Finally while we're on the topic: Does anyone know a good tool that can allow me to write an equation, specific the precision of inputs/outputs and get the required precision of all the operators? I know Matlab have the DSP toolbox but it has some serious limitations, I'm still in search of something fantastic.
- nestorD 8y ago
  Herbie is not an exact match for your need it might be useful nevertheless : https://herbie.uwplse.org/ https://herbie.uwplse.org/
man-and-laptop 8y ago
It's interesting to point out that you can compute with exact real numbers. But this is of course going to be less efficient than using something crude and fast like floating point numbers. There's also an argument that imprecision from your physical sensors is probably going to be more significant than the rounding errors that come with floating point. Apart from greater speed, floating point numbers don't really have any advantages over exact real numbers. Problems like the inability to compute discontinuous functions over the real numbers manifest as even worse problems over the floating point numbers. I'm also a believer that anything beautiful is probably going to end up useful. I think exact real computation (as described in Type Two Computability Theory) is one such thing. Automatic differentiation was another thing I found beautiful, before I found out about its applications to neural networks.
- sn41 8y ago
  Type-two theory has a huge disadvantage: it elevates the type of all operations by a level. For example, real numbers become functions. Hence addition becomes a type 2 functional. Integration becomes type 3. It is very hard to reason about the efficiency (and computability, come to think of it) of type 3 functionals. There's some esoteric example involving the fan functional [1]. The complexity theory of type-2 functionals is a theory which is natural to ask for, but after results by Cook, Bellantoni, Kapron, Seth, Pezzoli and others, has come to an impasse and now seems abandoned. [1] https://pdfs.semanticscholar.org/3888/cc2d3c44fb051067581e3947c3b9f9af1d5c.pdf https://pdfs.semanticscholar.org/3888/cc2d3c44fb051067581e39...
  - man-and-laptop 8y ago
    They effectively change the structure of your code. Every time you do an operation on a real number, it essentially launches a new process that runs indefinitely in the background. I would file that under speed problems. To control the speed penalty, you need to make deep changes to your code. I think Andrej Bauer came to this conclusion. Regarding computability, it's not so bad. Basically a continuous operator is usually computable. Example: Integration is continuous, hence (likely, and in this case actually) computable.
- 8y ago
sn41 8y ago
I understand floating point representation, but beyond that, it is a bit murky. I find floating point arithmetic to be very hard to really implement. For example, a basic question I have involves scaling: suppose I want to perform an operation involving large numbers like 13000.01 + 13000.01 + 13000.01. Do I get better precision results if I do 0.01 + 0.01 + 0.01 in floating point, then do the 3*13000 in integer arithmetic, and add the results?
- zamadatix 8y ago
  Significand * 2^exponent so yes. Another way to think of it more naturally is just remember it's a floating point in a fixed number of bits. Since base doesn't really matter for theory you can just think of a decimal point instead of binary. If you move it over to put more digits/information on the left side then you have to take digits/information from the right side. If you were going to do this though I'd just do 2 ints acting as a single fixed point. Less glue logic and likely more precision. A float using all of the bits would likely be better than int + float as well depending how you configured it.
- magicalhippo 8y ago
  A more general way to do that is Kahan summation[1], where you keep track of the rounding errors made so far. [1]: https://en.wikipedia.org/wiki/Kahan_summation_algorithm https://en.wikipedia.org/wiki/Kahan_summation_algorithm
sifoobar 8y ago
I've met plenty of coders who aren't even aware of the problem, and I spent 13 years maintaining and extending an accounting system written by people who couldn't tell a floating point issue from their ass on a good day. It's not going to happen much until we stop pushing floats as the standard solution for dealing with decimals. I'm pretty sure they are more commonly abused than used at this point, and most cases would be better off with fixpoints, rationals or bignums in increasing order of flexibility and complexity. Lisp and Perl6 get it more or less right. My own baby, Snigl [0], only supports integers and fixpoints so far. I might well add floats eventually, but I intend on hiding them well enough to not encourage anyone who isn't motivated. [0] https://gitlab.com/sifoo/snigl https://gitlab.com/sifoo/snigl
- ThePadawan 8y ago
  > (...) until we stop pushing floats as the standard solution for dealing with decimals. I'm curious, in that sentence, who is "we"? I cannot imagine a scenario where someone would be so technically involved to be aware of the difference between a floating-point and fixed-point decimal, but still decide to use floating-point in an accounting environment. Are you talking about project managers? Architects?
  - SpicyLemonZest 8y ago
    If you Google how to manipulate decimal values, or you type `rate = 1.23`, you're almost always going to get floating point. `1.1 + 2.2 == 3.3` yields false in most commonly used programming languages, with the notable exception of SQL. That's what it means that we've established floats as the standard. You have to specifically choose not to use floats, look up your language's entirely non-standardized approach for representing exact decimals, and then maintain constant vigilance against entirely intuitive things like decimal literals.
  - mdpopescu 8y ago
    Language designers, I would guess.
  - sifoobar 8y ago
porpoisely 8y ago
I doubt most developers really understand how floating points work ( as in how it is represented in memory and why and the mantissa, bias, etc ). Most understand that it is just an approximation and go on about their day.
- tracker1 8y ago
  When using floating point numbers, that's usually as far as you need to know. That and the range of safe integers for langauges like JS that only do float. (Though now there's BigInt at stage 3)
kazinator 8y ago
> Zero Divide By Zero: > This question asks whether in floating point 0.0/0.0 is a non-NaN value, which it is not. NaN generation is desirable here, since it will propagate to the output as a NaN and thus make the user suspicious. Horrible nonsense. What is desirable is an exception so that it's loudly confirmed to the user that something has screwed up. We don't want to lead the user on with subtle hints and suspicions. Even a dollar store calculator from 1985 gets this right by locking up with an E on he display, requiring a clear.
- jmalicki 8y ago
  This is configurable. Sometimes you want to see 99% of your results and know those were weird ones, and you can deal later.
hackermailman 8y ago
A lecture on IEEE floating point from CMU including most of the gotchas the pdf contains https://scs.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=7cceb0cb-e07c-4043-a7f3-d4d8c4661512 https://scs.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=... This is an active research area, ENS de Lyon has a lot of material available if interested implementing floating point on FPGAs, plus there is Vol 2 of The Art of Computer Programming where Knuth shows the reader how Babylonian mathematicians made extensive use of a sexagesimal (radix sixty) positional notation that was unique in that it was actually a floating point form of representation with exponents omitted.
stallmanifold 8y ago
How many software engineers had to study IEEE floating point in (any amount of) detail? Part of my formal education is in computer (hardware) engineering, and we studied hardware number representations and manipulations quite extensively in my microprocessors courses, so I developed some facility with the topic once upon a time.
cjhanks 8y ago
I have interviewed a lot of developers, and floating point precision is one of my common phone screen questions. My experience is; those who should know about it, do. Of course, a lot of apps or web developers don't. And I think that is okay. The bigger issue is code designed for float32 (for performance reasons) being assumed valid for float64 data. And in these cases the issue is always, what is the appropriate way to communicate to your user the intended use case of the function? When the writer is long gone, a user will not intuitively know. Whenever I have to use a 3rd party library (ahem, PCL). I peruse the code and grep for float variables... If I see too many, I simply assume I need to recenter my float64 to be near 0.0. Not everybody has the experience for this diligence, thus it appears as if developers don't understand.

116 points by matt_d 8y ago

Obligatory link to the classic paper on the subject, from 1991: https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.h...

gcc_programmer 8y ago

60% of the people who took the survey develop software to support their main role. So, 60% of the people are not really software engineers but coders by necessity: could this explain the results easily?

kbumsik 8y ago

Although understanding floating point is important, it is often ignored in Data Science schools. The title reminds me a friend of mine who attends a data science master course without computer background. He asked me for help for his numpy homework on Bayesian Probability theory. The number he got was so small that he got weird result. I immedieatly thought it was because float64 couldn't cover that number so I told him to use longdouble dtype (it's 80 bits in Intel I think) and the problem solved. Later he told me even the professor didn't expect such case when announcing the homework and many students couldn't finish the hw on time because of this.

de_watcher 8y ago
That's a difficulty that exists for the simulations, optimization problems and various algorithms. Some need so many bits that they don't work at all.
gronne 8y ago
No such thing as “Data science” - just a fancy term for trade school computer skills. Thats why.
- wolfgke 8y ago
  > No such thing as “Data science” - just a fancy term for trade school computer skills. Thats why. Relevant xkcd: https://xkcd.com/435/ https://xkcd.com/435/ Or as my friend put it: everything is just a trivial special case of either mathematics or physics. ;-)
  - gronne 8y ago
    True :) I know the strip. tbh I just felt like trolling the insecure “data hackers” crowd. Looking at downvotes: I think it worked.
  - everybodyknows 8y ago
    Fun xkcd, but mathematics does not belong on a continuum of the scientific disciplines, all of which seek to describe nature. "Die ganzen Zahlen hat der liebe Gott gemacht, alles andere ist Menschenwerk" https://en.m.wikipedia.org/wiki/Leopold_Kronecker https://en.m.wikipedia.org/wiki/Leopold_Kronecker Verbose, highly literal translation: "Loving God created the absolute integers; all else is the work of humanity."
mehrdadn 8y ago
Would the log-sum-exp trick have helped by any chance? https://en.wikipedia.org/wiki/LogSumExp https://en.wikipedia.org/wiki/LogSumExp
- kbumsik 8y ago
  I didn't know this (I'm not in ML/Data Science) but that looks very helpful. I looked through his learning materials but I have never seen this one though.
  - mehrdadn 8y ago
    Yeah I only happened to know because of ancourse I took that taught some ML. I mentioned it because, later on, I even came across a situation where it would've been directly applicable, but for some reason the thought that it might solve the problem escaped me at that moment, so I asked and someone else had to remind me of that fact... so I feel it's probably not the easiest thing to recognize when it's happening.
Veedrac 8y ago
> I immedieatly thought it was because float64 couldn't cover that number so I told him to use longdouble dtype (it's 80 bits in Intel I think) and the problem solved. This is a very dangerous habit. If 64 bits isn't enough you are generally using the wrong representation.

Upvoter33 8y ago

60% of the time, developers understand floating point 100% of the time.

caseymarquis 8y ago

I once encountered an old database using floats as its primary keys. This was in a fairly popular piece of manufacturing software from the 90s.

shabbyrobe 8y ago
Wow, I'd love to hear more about this.
mcv 8y ago
Useful in case you want multiple entries with an id of approximately 1?
- Const-me 8y ago
  Potentially, might be useful if you want small fixed-length keys that are fast to compare, and also you want to be able to insert new items between existing ones.
  - jmiserez 8y ago
    There was this article posted a while ago that used a Stern-Brocot tree and/or floats for ordering: https://news.ycombinator.com/item?id=16635440 https://news.ycombinator.com/item?id=16635440 It doesn't work in the pathological case though as any float has a fixed number of bits, and I see the article has been amended with a note.
maxxxxx 8y ago
What type of data would that be? I once did this when I did my own primitive version of a geospatial database in the 90s.
sevensor 8y ago
I sometimes, in Python, cautiously use floating point numbers as dictionary keys. I do this when what I mean is "this exact floating-point number" and not "the result of an equivalent computation." It's almost never a good idea, because it's really easy to trip yourself up by assuming the latter when only the former actually holds.

jstewartmobile 8y ago

Even if you understand them... in a polyglot world, you're still going to get burned. It is so easy--especially when you're jumping from a NUMBER language like SQL or Scheme to a floating-point one like C or JavaScript--to backslide into boners like for(i = 0.0; i < 1.0; i += 0.1) which would be a non-boner with a more advanced number tower. edit: My first Clojure program was the last time this got me--LISP on the outside, Java floats/ints/bigints on the inside.

heavenlyblue 8y ago
Yeah, and that’s why we have int for counting.
- jstewartmobile 8y ago
  So if I'm marking an axis from 0 to 0.9 in 0.1 increments, I should obviously do the integer indirection dance? After all, every damn fool knows--even ones like myself who've implemented FP in assembly--that 0.1 + 0.1 + 0.1 = 0.30000000000000004. Silly me!
  - heavenlyblue 8y ago
    >> So if I'm marking an axis from 0 to 0.9 in 0.1 increments, I should obviously do the integer indirection dance? Yes. Because you clearly don't understand what you're doing at all if you believe that equivalence relation is defined the same way for floating-points as it is defined for integers. Floating-points aren't numbers. They're more like confidence intervals - and at a certain limit a confidence interval behaves just like an integer. But it's only a limit. The fact that most users of programming languages don't want to keep track of that in their code only reinforces my belief that they must do exactly as the software guidelines tell them to.
    - jstewartmobile 8y ago
      I know what floating point is. I've implemented it in hardware. I've implemented it in software. But you know what? Floating point is not the only way people have ever represented real numbers on the computer. In most SQLs, decimal fixednum types are predominant. Most LISPs have a number tower that automatically juggles fixnums, bignums, ratios, floats, and complex numbers. In the 8/16-bit days--before everything had a floating-point unit--it was not uncommon to store and calculate numbers directly in BCD. Two-digits to the byte, and all of the operations--including formatting--were very straightforward to implement. Many of the old business mainframes even have hardware instructions to pack/unpack/operate on BCD. > that most users of programming languages don't want to keep track of that... Why would I want to keep track of that when the language (or more accurately, the data type) will do it for me? I'm sorry man. The whole world isn't C and JavaScript (yet). That we have forced crud-app developers to resort to garbage like this[0]--in places where the performance and storage benefits of floating-point are irrelevant--is a stain on the profession. [0] http://www.jacklmoore.com/notes/rounding-in-javascript/ http://www.jacklmoore.com/notes/rounding-in-javascript/

mysterydip 8y ago

If you're like me and struggled to understand the traditional textbook floating point formula explanation, Fabien Sanglard (of doom/wolf3d black book fame) wrote a page that finally made sense for me: http://fabiensanglard.net/floating_point_visually_explained/index.php http://fabiensanglard.net/floating_point_visually_explained/...

Waterluvian 8y ago
Can I ask an embarassing question to verify I'm not misunderstanding that page? Does this mean that floats are less precise the larger they are? You have as many bits to describe the "position of the number" between 0-1 as you do between 2048-4096
- Dylan16807 8y ago
  > You have as many bits to describe the "position of the number" between 0-1 as you do between 2048-4096 You mean 0.5-1 0-1 is about a quarter of the numbers, just as many as 1-inf
- pfortuny 8y ago
  They are less precise in the absolute sense (the absolute error is larger) but equally so in the relative sense (the relative error is the same). Well, essentially, of course.
- Sharlin 8y ago
  This is precisely what the term floating point means. The decimal point "floats" right or left depending on magnitude due to the mantissa-exponent representation. The opposite is fixed point where you have a fixed number of bits reserved for the whole and fractional parts. A 32-bit floating point can represent a vastly greater range of magnitudes than a 32-bit fixed point, with a vastly greater precision for values near zero, but at the expense of absolute precision at large magnitudes.
  - kps 8y ago
    At the computer floating point was developed, people understood it implicitly, because it was just an automated version of their current tools: log tables and slide rules. And likewise they knew better than to use floating-point where it doesn't work.

jmiserez 8y ago

Not developers. This survey included mostly PhD students (73), faculty (49), but only 23 software engineers. The rest are others. Only 1/4 see software engineering as their main role according to the paper. A PhD student is very different from a professional developer with years of real-world experience in industry. It's a completely different job. The title is highly misleading.

crispyambulance 8y ago
It's OK, not misleading at all, and the abstract spells out what they're studying and the motivation behind it clearly. A "developer" is just someone who writes software and not necessarily a "professional developer" with years of experience in a particular kind of job like what you have in mind. Moreover, I would even expect that the vast majority of "professional software engineers" who make a living strictly writing software without any particular domain in focus, might actually fare worse in floating point gotchas than scientific domain workers who are "software engineering dilettantes" but nonetheless do computation as their job.
- deleted 8y ago
  [deleted]
- jmiserez 8y ago
  Maybe. The paper even mentions this, sort of contradicting your expectation: >Perhaps not surprisingly, the most predictive factor is simply Contributed Codebase Size, the effect of which is shown in Figure 16. >There is a gain of only about 2/15 compared to those who reported a codebase where floating point was not intrinsic or where they were not involved. The title would be better suited if they included a higher number of HPC/ML/etc software engineers (post PhD) that work on larger scale projects in industry/labs. Specifically, I disagree with this assessment: >We believe that the combination of our recruitment process and the resulting background of the participants illustrated here suggest that our sample is a good representative of software developers who write code for, and in support of, science and engineering applications. That's just outright ignoring all science and engineering applications outside of academia/academic research.

abductee_hg 8y ago

according to this document the people who implemented java did not: "How Java’s Floating-Point Hurts Everyone Everywhere" https://people.eecs.berkeley.edu/~wkahan/JAVAhurt.pdf https://people.eecs.berkeley.edu/~wkahan/JAVAhurt.pdf p.s.: it's by William Kahan, who the primary architect behind the IEEE 754-1985 standard for floating-point computation (and its radix-independent follow-on, IEEE 854)

nraynaud 8y ago
no language has a good support for IEEE754. The only doable thing is using some weird macros in C/C++, but the rest of the languages ignore the issue altogether. And LLVM too.
- ChrisLomont 8y ago
  C/C++ is not required to use ieee754 as it's float format, so your code would always run under that cloud. Fortran, D, Factor, and SBCL likely have better support. But anything where a compiler can reorder things is suspect, if you need exact behavior, since even simple register spilling causes trouble (Intel), among other things compilers may do.
  - Asooka 8y ago
    AFAIK C and C++ compilers are not allowed to re-order floating-point arithmetic precisely for that reason. Also, given that all cpus support SSE/Neon/etc. now and I haven't seen a compiler use the 387 FPU, I suspect register spilling shouldn't be an issue these days. Of course, legacy/exotic platforms will deviate from the norm in fun ways.
    - ChrisLomont 8y ago
      >AFAIK C and C++ compilers are not allowed to re-order floating-point arithmetic precisely for that reason. You don't have to reorder to make bugs. Simply having a compiler replace (a-b)+b with a is a bug. Simply flushing an internal 80-bit floating register to a 64 bit memory place, then loading it, causes differences in computations. Note the C++ spec specifically states it does not require an underlying representation or even any certain level of accuracy (6.7.1.8). So many of these bugs are allowed. The spec does not even require that compile time evaluation of an expression must match a runtime evaluation of the same expression (8.20.6). When you hit that in code you're going to be surprised. C/C++ is not required to round-trip floats/doubles when printed and parsed. stream formatting rounding modes are allowed (and are) not exact, leading to different behavior on formatting the same number on different systems. Dealing with these led to compiler implementation defined behavior added due to market pressures, making C/C++ numerics less of a mess, not anything in the spec. C/C++ has notoriously been bad at all this, which is why the history of it is littered with such bugs, errors, undefined behaviors, and dozens of compiler switches to try and mitigate such behavior. For example, many compilers notoriously replaced high-precision Kahan summation with low-precision regular summation because optimizations tell the compiler that such things are allowed. The state of the art in C/C++ is now passable, but it is still flawed, and has been historically very bad. And all this met the C/C++ spec of the time. Because if the underlying representation does not require IEEE 754, then it cannot require only transformations for IEEE 754 without making other representations behave badly.

known 8y ago

Yes with http://0.30000000000000004.com/ http://0.30000000000000004.com/

Traster 8y ago

This is a pet subject of mine since as an FPGA developer a huge amount of my time is spent focusing on bit-precision and Fixed-Point and reduced precision floating point implementations of algorithms. In fact one of the interview questions I ask focuses on how to do different mathematical operations in floating point using only logical/integer operations. On the one hand I find the results unsurprising - even people I know to have worked a lot on numerics often have only a rudimental understanding of the intricacies of corner case behaviour for floating point, and yes that absolutely manifests in hitting a wall when something curious goes wrong. Mostly this results in head banging until you find the piece of code going wrong is the numerical piece and then very quickly you start looking at possible floating point gremlins. Having said that though this paper seems to have a very academic view of what HPC is. Even for people designing HPC systems numerical optimizations are rarely a huge chunk of their job so it's probably not actually that important - I think the fact few people have a good understanding is a reflection of the fact it's not often necessary. Finally while we're on the topic: Does anyone know a good tool that can allow me to write an equation, specific the precision of inputs/outputs and get the required precision of all the operators? I know Matlab have the DSP toolbox but it has some serious limitations, I'm still in search of something fantastic.

nestorD 8y ago
Herbie is not an exact match for your need it might be useful nevertheless : https://herbie.uwplse.org/ https://herbie.uwplse.org/

man-and-laptop 8y ago

It's interesting to point out that you can compute with exact real numbers. But this is of course going to be less efficient than using something crude and fast like floating point numbers. There's also an argument that imprecision from your physical sensors is probably going to be more significant than the rounding errors that come with floating point. Apart from greater speed, floating point numbers don't really have any advantages over exact real numbers. Problems like the inability to compute discontinuous functions over the real numbers manifest as even worse problems over the floating point numbers. I'm also a believer that anything beautiful is probably going to end up useful. I think exact real computation (as described in Type Two Computability Theory) is one such thing. Automatic differentiation was another thing I found beautiful, before I found out about its applications to neural networks.

sn41 8y ago
Type-two theory has a huge disadvantage: it elevates the type of all operations by a level. For example, real numbers become functions. Hence addition becomes a type 2 functional. Integration becomes type 3. It is very hard to reason about the efficiency (and computability, come to think of it) of type 3 functionals. There's some esoteric example involving the fan functional [1]. The complexity theory of type-2 functionals is a theory which is natural to ask for, but after results by Cook, Bellantoni, Kapron, Seth, Pezzoli and others, has come to an impasse and now seems abandoned. [1] https://pdfs.semanticscholar.org/3888/cc2d3c44fb051067581e3947c3b9f9af1d5c.pdf https://pdfs.semanticscholar.org/3888/cc2d3c44fb051067581e39...
- man-and-laptop 8y ago
  They effectively change the structure of your code. Every time you do an operation on a real number, it essentially launches a new process that runs indefinitely in the background. I would file that under speed problems. To control the speed penalty, you need to make deep changes to your code. I think Andrej Bauer came to this conclusion. Regarding computability, it's not so bad. Basically a continuous operator is usually computable. Example: Integration is continuous, hence (likely, and in this case actually) computable.
8y ago

sn41 8y ago

I understand floating point representation, but beyond that, it is a bit murky. I find floating point arithmetic to be very hard to really implement. For example, a basic question I have involves scaling: suppose I want to perform an operation involving large numbers like 13000.01 + 13000.01 + 13000.01. Do I get better precision results if I do 0.01 + 0.01 + 0.01 in floating point, then do the 3*13000 in integer arithmetic, and add the results?

zamadatix 8y ago
Significand * 2^exponent so yes. Another way to think of it more naturally is just remember it's a floating point in a fixed number of bits. Since base doesn't really matter for theory you can just think of a decimal point instead of binary. If you move it over to put more digits/information on the left side then you have to take digits/information from the right side. If you were going to do this though I'd just do 2 ints acting as a single fixed point. Less glue logic and likely more precision. A float using all of the bits would likely be better than int + float as well depending how you configured it.
magicalhippo 8y ago
A more general way to do that is Kahan summation[1], where you keep track of the rounding errors made so far. [1]: https://en.wikipedia.org/wiki/Kahan_summation_algorithm https://en.wikipedia.org/wiki/Kahan_summation_algorithm

sifoobar 8y ago

I've met plenty of coders who aren't even aware of the problem, and I spent 13 years maintaining and extending an accounting system written by people who couldn't tell a floating point issue from their ass on a good day. It's not going to happen much until we stop pushing floats as the standard solution for dealing with decimals. I'm pretty sure they are more commonly abused than used at this point, and most cases would be better off with fixpoints, rationals or bignums in increasing order of flexibility and complexity. Lisp and Perl6 get it more or less right. My own baby, Snigl [0], only supports integers and fixpoints so far. I might well add floats eventually, but I intend on hiding them well enough to not encourage anyone who isn't motivated. [0] https://gitlab.com/sifoo/snigl https://gitlab.com/sifoo/snigl

ThePadawan 8y ago
> (...) until we stop pushing floats as the standard solution for dealing with decimals. I'm curious, in that sentence, who is "we"? I cannot imagine a scenario where someone would be so technically involved to be aware of the difference between a floating-point and fixed-point decimal, but still decide to use floating-point in an accounting environment. Are you talking about project managers? Architects?
- SpicyLemonZest 8y ago
  If you Google how to manipulate decimal values, or you type `rate = 1.23`, you're almost always going to get floating point. `1.1 + 2.2 == 3.3` yields false in most commonly used programming languages, with the notable exception of SQL. That's what it means that we've established floats as the standard. You have to specifically choose not to use floats, look up your language's entirely non-standardized approach for representing exact decimals, and then maintain constant vigilance against entirely intuitive things like decimal literals.
- mdpopescu 8y ago
  Language designers, I would guess.
- sifoobar 8y ago

porpoisely 8y ago

I doubt most developers really understand how floating points work ( as in how it is represented in memory and why and the mantissa, bias, etc ). Most understand that it is just an approximation and go on about their day.

tracker1 8y ago
When using floating point numbers, that's usually as far as you need to know. That and the range of safe integers for langauges like JS that only do float. (Though now there's BigInt at stage 3)

kazinator 8y ago

> Zero Divide By Zero: > This question asks whether in floating point 0.0/0.0 is a non-NaN value, which it is not. NaN generation is desirable here, since it will propagate to the output as a NaN and thus make the user suspicious. Horrible nonsense. What is desirable is an exception so that it's loudly confirmed to the user that something has screwed up. We don't want to lead the user on with subtle hints and suspicions. Even a dollar store calculator from 1985 gets this right by locking up with an E on he display, requiring a clear.

jmalicki 8y ago
This is configurable. Sometimes you want to see 99% of your results and know those were weird ones, and you can deal later.

hackermailman 8y ago

A lecture on IEEE floating point from CMU including most of the gotchas the pdf contains https://scs.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=7cceb0cb-e07c-4043-a7f3-d4d8c4661512 https://scs.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=... This is an active research area, ENS de Lyon has a lot of material available if interested implementing floating point on FPGAs, plus there is Vol 2 of The Art of Computer Programming where Knuth shows the reader how Babylonian mathematicians made extensive use of a sexagesimal (radix sixty) positional notation that was unique in that it was actually a floating point form of representation with exponents omitted.

stallmanifold 8y ago

How many software engineers had to study IEEE floating point in (any amount of) detail? Part of my formal education is in computer (hardware) engineering, and we studied hardware number representations and manipulations quite extensively in my microprocessors courses, so I developed some facility with the topic once upon a time.

cjhanks 8y ago

I have interviewed a lot of developers, and floating point precision is one of my common phone screen questions. My experience is; those who should know about it, do. Of course, a lot of apps or web developers don't. And I think that is okay. The bigger issue is code designed for float32 (for performance reasons) being assumed valid for float64 data. And in these cases the issue is always, what is the appropriate way to communicate to your user the intended use case of the function? When the writer is long gone, a user will not intuitively know. Whenever I have to use a 3rd party library (ahem, PCL). I peruse the code and grep for float variables... If I see too many, I simply assume I need to recenter my float64 to be near 0.0. Not everybody has the experience for this diligence, thus it appears as if developers don't understand.