6 ms·
How to defeat the purpose of IEEE floating point (2008)
- markrages 12y agoThe quest for consistency seems misguided. If your code changes behavior based on the timing of the various sections, you have a bug, a race condition. The solution isn't to lock down your code and build with only one configuration. The solution is to fix the bug. In the same way, if the behavior of your code is dependent on the order of operations in floating point, it is very likely that you are doing something wrong. The solution is not careful ordering of floating point operations, but using algorithms appropriate to floating-point (and avoiding floating point when it isn't applicable.) Code that has been ported to many platforms tends to be more stable, because the heterogeneous environments have exposed bugs and flawed assumptions in the code's construction.
- lmm 12y agoSimple use case: a multiplayer game containing physics. I'd like the physics to be reasonably accurate, but more than anything it needs to be consistent, because I want to get the same results on both players' machines, and then the only thing I have to send over the slow internet is a command stream.
- markrages 12y agoBut if infinitesimal disturbances to the results makes a practical difference to the game, then your physics engine is not stable.
- Terr_ 12y ago> But if infinitesimal disturbances to the results makes a practical difference to the game They don't! The system can tolerate answers which are slightly wrong... as long as all machines calculate the SAME slightly-wrong answer. The same logic applies to synchronized PRNGs. It doesn't (directly) matter what the "random" dice-roll result is, as long as all clients get the same answer.
- markrages 12y agoAh, thank you for this response. It "clicked" for me: Game developers are using the low bits (below significance) as pseudorandom numbers! They are overloading a new function on top of the existing use of floating-point numbers.
- Terr_ 12y agoThat's not what I'm saying. I'm just trying to give you an example where a "correct result" is far less-important than a "deterministic result".
- tomlu 12y agoNo physics engine is stable in that sense. Neither is the real world. You need full determinism to do what lmm is saying. Source: Worked on this exact thing for commercial multiplayer games.
- teraflop 12y agoMany interesting physical phenomena are deterministic but chaotic, in the sense that arbitrarily tiny changes in initial conditions can rapidly lead to arbitrarily large changes in behavior. Would you say that anyone who tries to make a game that has the same property is "doing something wrong"?
- Scramblejams 12y agoYou're probably imagining a short-lived game. Think of a game that goes on for months at a time. There are ways around this, but none of them as elegant as a floating point routine which produces the same results on any platform across any ordering of calculations.
- jephir 12y agoEven for short-lived games, if you want to implement a replay system, you need your physics to be deterministic.
- TeMPOraL 12y agoIt's a feature of physics, not physics engines. Many process in the real world are exponential in nature, so even a very small difference gets magnified very quickly.
- idlewords 12y agoI read the article so you don't have to! The author dreams of consistent behavior so he can get: - Reproducible results across platforms - Reproducible results with different build settings (such as debug builds or optimization levels) - Useful test cases for regression testing - Test cases that help him find misbehavior in the optimizer - An improved chance of finding more subtle bugs, absent 'noise' from small changes in floating point operations In particular, he gives examples of how changes to nearby code can change the output of floating point operations, due to hairiness in how registers are allocated. You can see how this would make testing quite painful. These all seem like worthy goals that deserve a better response than "you're doing it wrong", along with some non-sequitur about race conditions.
- markrages 12y agoRegarding race conditions, I was referring to this: > It's not just floating point that's inconsistent across modes – it's code snippets with behavior undefined by the language, buggy dependence on timing, optimizer bugs, conditional compilation, etc.
- pjc50 12y agoThere's quite a few algorithms that are generally vulnerable to precision issues because they can affect comparison order, such as k-means clustering. "Don't do that then" is not a useful answer.
- marcosdumay 12y agoLots of problems dictate code that depend on the ordering of floating point operations. It's not something that you can design away.
- lmm 12y agoThis kind of thing makes me happy I work on the JVM. strictfp may be "cowardly" but it's relatively easy to use.
- Terr_ 12y agoI understand that avoids some platform-capability differences, but does it also protect from issues with execution-order or ghostly-coupling between threads?
- tomlu 12y agoI have never heard of those problems, what are they? Hardware bugs?
- pcwalton 12y agoIn C terms strictfp just sets FLT_EVAL_METHOD to 0, which in today's SSE and 64-bit world most C compilers do anyway.
- shawnhermans 12y agoMaybe I am wrong, but is the author just looking for a decimal type? Those exist in most major languages, right?
- wolf550e 12y agoThe performance of decimal arithmetic on hardware not recently made by IBM means it cannot be used in most places.
- maxlybbert 12y agoThe original post said he'd be happy to trade performance for consistency. He could also use fdlibm ( http://netlib.org/fdlibm/index.html http://netlib.org/fdlibm/index.html ), at least for anything other than primitive operations. But, again, it trades performance for consistency.
- maxlybbert 12y agoThis week I learned that C99 and C11 have an FP_CONTRACT pragma, that can be set "off" for this reason ( http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf , section 6.5, paragraph 8).
- TheLoneWolfling 12y agoDecimal types suffer from many of the same issues. Sure, 0.1 can be represented exactly. But that's about the only issue with floating-point that that solves.
- srean 12y agoI have often found myself closer to the opposite end of the spectrum. Many machine learning algorithms and applications (certainly not all) can tolerate a fair degree of inaccuracy. This is often justified by the fact that the training data is typically noisy to begin with and is a random sample from the space it lives in. If your algorithm can be stable under those conditions it ought to be stable under floating point jitter. This usually means you need regularization, or smoothing, or low pass filter, or averaging in some guise or the other (in some abstract way they are the same thing). There are some frequently used operations in ML that are very unstable under inaccuracies, numeric differentiation is one, just dont do it without smoothening. Inverting matrices is another, but you better have a real solid reason to ever invert a matrix, yes there are reasons when you do really want the inverse. In any case the final point is that I have had to baby talk the compilers into, "yes, please please dont worry about it, just do these commutative (for real numbers) operations in any order that you want". This exposes more parallelism and other compiler optimizations. Often I would gladly take the hit over some precision in higher order decimal places to gain speed. But this should not be done blindly, and a smattering of numerical analysis helps gauging when it would be safe to do so. I doubt whether numerical analysis figures prominently in the trajectory of a graduating machine learner, although I think it ought to.
- TheLoneWolfling 12y agoOne of the advantages of languages with N-ary operators. * (a,b,c) can take advantage of those tricks, whereas the programmer can still be explicit and go * (a, * (b, c)), for example.
- srean 12y agoIndeed. Do you have some specific examples in mind, I would love knowing more. The usual suspects, for example fold_left, fold_right would typically adhere to fixed order for floating point operations. A reason Java is a FLOPs killer is that its standard is too strictly specified, it does not leave much room for the runtime to take advantage of these features. Even order of argument evaluation for functions are specified to be left to right.
- jephir 12y agoIf you're a game developer, you should look at this Floating Point Determinism article, which shows some additional techniques for implementing deterministic physics with floating point. http://gafferongames.com/networking-for-game-programmers/floating-point-determinism/ http://gafferongames.com/networking-for-game-programmers/flo...
- rurban 12y agoGetting consistently worse results? :) He didn't even mention when matlab 5 started using his 64b only recommendation, the results were getting in-precise and matlab was getting unusable. http://www.cs.berkeley.edu/~wkahan/MxMulEps.pdf http://www.cs.berkeley.edu/~wkahan/MxMulEps.pdf I cannot take a floating point rant seriously without knowing Kahan's arguments. And a crazy requirement to compare it to slow and false debugging code does not really help.