4 ms·
Anyone care to explain the details on a Texas Instruments NSpire calculator (or CAS)? a:=0.1+0.2 b:=0.3 a-b |> 0. Using the non-exact mode.
by soegaard 11y ago
Anyone care to explain the details on a Texas Instruments NSpire calculator (or CAS)?
a:=0.1+0.2
b:=0.3
a-b |> 0.
Using the non-exact mode.
- acqq 11y agoCalculators have traditionally used base 10 internally, not base 2, what is the base of the numbers on that TI? That's the answer, if it's base 10, .1, .2 and .3 are all exact.
- acqq 11y agoThe Wikipedia mentions base 10 with TI Basic: https://en.wikipedia.org/wiki/TI-BASIC https://en.wikipedia.org/wiki/TI-BASIC "Real numbers, using decimal floating point. These store up to 14 significant digits depending on the calculator model."
- soegaard 11y agoThis is it. An example: (1/3)*3.-1 gives -1E-14
- jwmerrill 11y agoJust a guess, but they're probably cheating the display. Try (a-b)*10^16
- soegaard 11y agoI tried some variations before I posted, but the result was just 0.