5 ms·
It's not "round off error", though. More like: "It's a base 2 fractional number with no exact decimal expansion with a finite number of digits. Display it in
by caf 16y ago
It's not "round off error", though. More like:
"It's a base 2 fractional number with no exact decimal expansion with a finite number of digits. Display it in base 2 fractional form if you don't want to see an approximation."
- aliguori 16y agoWell, it's more like: Binary numbers are a countably infinite set. Decimal numbers are a countably infinite set. You can therefore map binary representation to decimal representation 1-1 with a simple mapping function. OTOH, floating point numbers are an uncountably infinite set. The only way to map binary numbers to floating point numbers is to map to a strict subset. The subset happens to be different when mapping binary to floating point and decimal to floating point when using IEEE754.
- mseebach 16y agoThe mathematician: That's fine for 22/7. Now, go fix your computer so it knows how to count to 0.3, kthxbye
- eru 16y agoAs a mathematician I do not value 0.3 above 22/7.
- caf 16y ago0.3 is a rational too, it's just 3/10. Anyway, mathematicians don't use numbers at all - debasing the equations by performing caculations with them is so.. gauche. Leave that to the physicists, chemists and engineers.
- eru 16y agoI saw the occasional number in my studies. We'd rather calculate with knows than numbers. (See http://en.wikipedia.org/wiki/Knot_theory http://en.wikipedia.org/wiki/Knot_theory)
- messel 16y agoNo doubt, for years I've called quantization errors round off errors. But there's no rounding, it's an approximation error.