17 ms·
I feel like floating point is a completely separate branch of programming that a lot of coders never use. When they do it's often a mistake, like for currency.
by TorKlingberg 6y ago
I feel like floating point is a completely separate branch of programming that a lot of coders never use. When they do it's often a mistake, like for currency. Yet floating point is very useful for scientific calculations and simulations, which is what computers were all about for the first couple of decades.
I have a suspicion that on a fundamental level, floating point isn't actually good for games or machine learning. They're just used because existing computers are so good at floating point number crunching, especially GPUs.
- linkdd 6y agoIn some game engines, Fixed Point Arithmetic is used instead of Floating Point, because it's faster, and the approximation is generally good enough for a game.
- quelsolaar 6y agoI dont think this is true. Integer math is used in some game engines because they need to be deterministic, between networked computers (and different CPUs rounds floating point numbers differently). I have written an engine like that, and i know that Starcraft 2 is all integer for the same reason. No one does it because its faster or easier. Its a pain.
- linkdd 6y agoI never said it was easier and I never said ALL game engines. And computing integers IS faster than computing floats, at least on a CPU.
- quelsolaar 6y ago>And computing integers IS faster than computing floats, at least on a CPU. In the abstract yes. In reality not so much. A lot of what floats are used for in games is vector math (you are often in a 2D/3D world), and vector math in fixed point requires a lot more work in integer math since you constantly need to shift down things in order to avoid overflow. Overflow bugs are a constant problem when doing dot products in fixed point integer math. Another common operation in vector math is square root (to normalize vectors) Modern CPU's do that very fast, and you can also use clever approximations that use the way floating point's are represented in hardware. Most of whats done in 32 bit floats in games, needs to be done with 64bit integers to manage these problems if you decide to use integers, and that means that your data grows, more cache misses, and things slow down. On top of this everything you do in a game needs to be drawn on the GPU so you need to convert everything to floats anyway to show them on screen, and converting between integers and floats is also slow.
- linkdd 6y agoI never use the square root in games, I always compare squared distances (since the properties are the same). If you try to do with Fixed Point Arithmetic what was intended to be done with Floating Point, you're the problem. A 32 bits integer can hold values up to 4 billions, if I have that kind of value in a simple game, then yes i will switch to Floating Point Arithmetic, but when does that use case happen if you're not writing a physic simulation game ?
- adwn 6y agoIt appears you have a very niche definition of "game", one which excludes basically all 3D games.
- linkdd 6y agoYes, that's exactly why I wrote: > some game engines and > simple games A 3D game is not a simple game.
- adwn 6y agoUgh, do you really have to be this nitpicky? Fine, you wrote: > the approximation is generally good enough for a game No, it's not generally good enough for a game – only for some games, and not good enough for any game with 3D graphics.
- Qwertious 6y agoSpeaking of nitpicky, I thought the gamecube's GPU was integer-driven? And it has plenty of 3D games.
- yvt 6y agoTechnically, 32-bit fixed-point numbers have 8-bit more precision compared to 32-bit floating-point numbers, but the real problem is it's quite cumbersome to fully utilize the precision offered by fixed-point numbers as you'd have to keep in mind their numerical range all the time.
- namibj 6y agoModern CPUs, like Zen2, have vastly more f64 multiplier throughput than 64 bit integer multiplier throughput. This goes so far, that Intel added an AVX512 instruction to expose this 52/53 bit multiplier for integer operations, to accelerate bigint cryptography calculations like RSA and ECC.
- sandworm101 6y agoIt depends on which 0 you are talking about. The 1-1=0 or the 1/infinity zero. Somewhere deep in a physics department there is an equation where the distinction either confirms or denies the multiverse.
- justincredible 6y ago1/infinity is not 0, it's infinitesimal.
- bjourne 6y agoWhich modern general-purpose CPUs do not follow IEEE 754? I don't think what you're saying is true.
- deleted 6y ago[deleted]
- quelsolaar 6y agoThey all follow IEEE 754. But that defines the representation of the bits in a floating point number, not what happens when you operate on them. Different implementations do rounding differently, and there are arguments about how to do this correctly. There can be multiple ways to represent the same number even (this is a reason not to ever do a == compare on floating point number that have been operated on). Not only have there been differences between CPU makers, but also between different CPU of the same vendor. A separate, complicating factor is that floating point math often don't yield the same results with different levels of optimization turned on. The reason to use integer math for games is because you want it to be deterministic so can be an issue.
- zokier 6y agoIEEE 754 does also specify operations and their behavior. E.g. section 5.1 (from 2008 edition): > All conforming implementations of this standard shall provide the operations listed in this clause for all supported arithmetic formats, except as stated below. Each of the computational operations that return a numeric result specified by this standard shall be performed as if it first produced an intermediate result correct to infinite precision and with unbounded range, and then rounded that intermediate result, if necessary, to fit in the destination’s format (see 4 and 7).
- quelsolaar 6y agoWell the fact is that different hardware rounds it differently, so its the world we live in. My guess is that actually implementing something in hardware " to infinite precision and with unbounded range" might not be realistic.
- neurocline 6y agoStarcraft II uses floats, but for the game simulation portion it uses an internal software floating-point library to ensure consistency.
- quelsolaar 6y agoYou are right, all the graphics are full of floats. Writing "all" was a bit of an overreach.
- namibj 6y agoActually, the rounding for basic add/multiply are strictly defined and configurable (some algorithms need other rounding rules than the default "round to nearest even"). If you only use those (maybe including division, I'm not sure), and don't rely on hardware support for sqrt, sin/cos/tan, approximate-inverse, etc., you can totally use them deterministically between different architectures.
- maximilianburke 6y agoIt might be a little faster for addition and subtraction in synthetic benchmarks -- and with the use of pipelining might end up being slower overall -- but multiplication and division are considerably faster with floating point.
- dnautics 6y agodivision, yes, but multiplication is faster, but it's not huge. The limiting factor for multiplication is that it's O(N^2) in multiplied digits; so a 32-bit fixed point has 32x32 and IEEE fp32 multiplication has 23x23.
- krackers 6y agoThat's assuming that you're implementing multiplication in software though, right? Since CPUs have dedicated multipliers, aren't both around a single cycle?
- dnautics 6y agoThey are not in general (iirc 64b mult is down to 3 cycles now), but usually in an x86 cpu it can run microinstructions out of order enough so that the multiplication looks like it's finished in one cycle.
- quelsolaar 6y agoAs someone who has started out making games on Playstation 1 that didn't have floating point hardware, I can with authority say that developing games with only integers suck. Floating point number are far easier and robust to use.
- pfraze 6y agoOnly integers? Wow. Were int pairs used to represent numbers with decimals or what?
- timerol 6y agoTo use only integers, you generally try to express all numbers in units of the smallest quantity. So any money is in integer cents, any distance is in integer (spatial resolution size). It gets annoying with derived units: Time can be stored in integer frames, but then you'll need a rational type of some kind to properly handle velocity. Or you just round to the nearest (spatial resolution) per frame and hope the errors don't look too weird on the other side.
- salicideblock 6y agoIn signal processing on constrained domains (constrained including the lack of an FPU), the usual alternative to floating point is "fixed point" arithmethic. Basically you use the platform's native types (e.g. uint32) and decide which of the bits are for the integer part and which are for the fractional. uint32 can be interpreted as 20/12 for example. Your register went from representing "units" to representing "2^-12 increments of an unit". The ALU can do sum/comparison/subtractions transparently, but multiplications and (god forbid) divisions require shifting to return to the original representation. The choice of how to split the bits is a tradeoff between range and precision. It can vary from one routine to another. It's a pain to compose code with it. Short story: native float types are a blessing for programmer productivity and library interoperability.
- okl 6y agoThe Wikipedia page explains the operations: https://en.wikipedia.org/wiki/Q_(number_format) https://en.wikipedia.org/wiki/Q_(number_format)
- legulere 6y agoOne example where floating point does not match the problem field is GIS. Coordinates are bound by the earth's circumference. Fixed point would make much more sense, but neither hardware nor software support is there.
- lylecheatham 6y agoI mean most GIS software is pretty aware of precision models. Take GEOS for example, it's got a variety of ways to specify the precision in number of decimal places for a geometry. It still uses doubles underneath, but that's to avoid unneeded code complexity.
- oivey 6y agoLots of the math in games and ML assumes you’re operating in the reals. For example, the optimization in neural networks. Integers and fixed point won’t work.
- bwestergard 6y agoFloats have a totally different algebra than the reals, though.
- adwn 6y agoOnly in the strictly mathematical sense. To a first approximation, floating-point numbers behave like real numbers, and that's good enough for many use-cases (not all, though). Similarly, fixed-width integers have a totally different algebra than true integers, and yet they're immensely useful.
- legulere 6y agofixed-width integers have exactly the same semantics as true integers bar overflow, which is a very easy to understand concept. Floating-point numbers behave like a physicist would do calculations rounding at every step, but doing everything in binary. Or hexadecimal if you prefer, it's equivalent, but still difficult to get your head around to. Then there additionally is + and - 0 and + and - infinity, and several settings for how rounding is done exactly.
- bwestergard 6y ago"To a first approximation, floating-point numbers behave like real numbers" This is incorrect. To count as an approximation, there have to be some accuracy bounds. This is impossible to define, as the reals don't have the same cardinality as any floating point number system. Now, for many interesting and useful real-valued functions, a float-valued function can be defined that is an approximation. But there is no general way to map a function defined on the reals to a function defined on the floats with a known accuracy.
- oivey 6y ago
- TacticalCoder 6y ago> When they do it's often a mistake, like for currency. The most beautiful FP bug I remember was a denial of service in some webservers where by setting the header "Accepted Language: en-gb;q=0.3 en;q=0.8 ..." to a specific value you could send the official Java floating-point parser in an infinite loop (and this affected several Java webservers). So at each webpage request, you were sending one CPU core of the webserver basically busy-looping. Hence crashing the webserver in a few requests at most. Now it'd be heresy if I were to say that maybe, just maybe, had the standards mandated to use 0 to 100 for weighting instead of 0 to 1.0 we could have dodged a whole source of potential issues!? No, no, I realize this is heresy: let's all keep using numbers that cannot even be represented correctly (except as strings), parse those strings into something approximately representing what's written in the string, then make more approximation errors while doing computation with these numbers, propagation errors, epsilon estimation errors, and keep insisting we all could have authored "What every computer scientist should known about floating-point numbers" (which is 80 pages long and, well, approaching a treaty) ; ) /rant off
- strictfp 6y agoMeh, it's not like there can't be a bug in whatever other parsing code a webserver uses. Maybe using a float is overkill in this case, since you're never going to hit more than a hundred languages. But it's at least not setting any limitations.
- mattgreenrocks 6y agoThe issue is that the representation of floating point numbers imposes additional cognitive burden that the code must handle. How does the web server header parsing code handle positive/negative infinity? How about NaN? NaNs/infs are particularly vicious, as they propagate through computations.
- rsj_hn 6y agoThis. Floating point numbers, as they are lossy attempts to discretize continuous data, are more complex and usually non-deterministic. They should never be chosen when a non-lossy representation is available for the problem domain.
- civilized 6y agoFloats are essential for machine learning, scientific computing, and any other application where the goal is to model mathematical processes involving real numbers. The idea that there is something out there that could do the job better than floats is almost certainly wrong. I'm fascinated by how common it is for programmers to hate floats. Yes, if you write business software for a living you may not have much use for them. But there's a lot more to computing than business software.
- amelius 6y ago> Floats are essential for Not necessarily true. Theoretically you can replace floats by fixed-point representations given enough bits, and in practice this also works often. It's a pity languages/hardware don't have good built-in support for fixed-point numbers though.
- lostcolony 6y agoWell, as mentioned elsewhere, because of historical purposes, GPUs tend to optimize floating point operations really heavily. Whereas bigint style things likely don't benefit, in part because there's no "standard" for them at that low a level. So it seems to me to be a bit of chicken and egg; low level languages don't have them as part of the standard, so graphic cards can't universally optimize for them, and since graphic cards don't support optimizations for them, there isn't anyone clamoring for it.
- civilized 6y agoOkay, so now, relative to floating point arithmetic, you've sacrificed an enormous amount of dynamic range and have to worry about overflow and underflow much more than before. What did you gain from this, exactly? I'm sure there are some special situations where fixed point is better, but for general purpose scientific and technical computing, floating point is obviously the right choice.
- deleted 6y ago[deleted]
- emiliobumachar 6y ago> very useful for scientific calculations and simulations Don't forget control systems.
- PaulHoule 6y agoIt is the opposite. From Microsoft BASIC to Javascript, many programmers have worked in languages that only have floats. Financial calculations involve exponential/log math much like scientific problems. There are issues with rounding and representation there (e.g. there is no such thing as $0.01 in floating point, only $0.50, $0.25, $0.125 and some sum of 1/2 fractions that comes very close to $0.01 and even reads in and writes out as $0.01 even though it isn't.)
- kingkonz1 6y agoWhat better alternative is there for handling currencies? Multiplying the original amount by 100 and casting to an int? ($3.50 -> 350)?
- shp0ngle 6y agoIn javascript, every number is technically floating point. :)