3 ms·
15 decimal digits around is the approximate limit for a 64-bit IEEE floating-point number, which is what the built-in Calculator uses (most non-integral math do
by eridius 11y ago
15 decimal digits around is the approximate limit for a 64-bit IEEE floating-point number, which is what the built-in Calculator uses (most non-integral math done on computers uses either 32-bit of 64-bit IEEE floating-point numbers). The exponential notation is because floating-point numbers can represent absurdly large values, it just is limited on precision. So you can represent 10^31, you just can't represent 10^31 + 1.
- sneak 11y agoWhy should the user of a calculator care how the numbers are represented internally? More importantly, why should they be forced to? It's a bug.
- 0xdeadbeefbabe 11y agoWell, what if these users you speak of are trying to accomplish more than just homework?
- cbhl 11y agoSome users do expect arbitrary precision for basic operations (addition, subtraction, multiplication, division) since the Calculator app that comes with Windows does it[0]. [0] http://blog.codinghorror.com/if-you-dont-change-the-ui-nobody-notices/ http://blog.codinghorror.com/if-you-dont-change-the-ui-nobod...