3 ms·
It takes some time to learn how floating point and numerical calculations work. Not too long, but more than one evening. If you take a numerical analysis course
by sampo 5y ago
It takes some time to learn how floating point and numerical calculations work. Not too long, but more than one evening. If you take a numerical analysis course in a university, the first 2 or 3 lectures might be about floating point, error propagation and error analysis. Or the first chapter of a numerical analysis textbook.
But almost nobody spends this much effort to familiarize themselves with the floating point system. So it keeps surprising people.
- josefx 5y agoAt least it should be a bit less surprising now that every CPU has SSE instructions where a 32 bit float is actually stored in a 32 bit vector register instead of a 80 bit FPU register that may be dropped to a 32 bit memory location whenever your compiler feels like it. Makes the results a lot more consistent.