3 ms·

How do Sheets and Excel differ in this regard? How did using floats cause some number to be odd?

• jdmichal 4y ago
  GP didn't say odd, or at least not numerically odd, but rather "ending in impossible digits". Floats have varying precision, not uniform. The closer the number is to zero, the more precision it has. There's a point where the precision is so low that it only covers whole numbers, and then a point after that where the precision is less than whole numbers. Given the context of pointers, it's quite possible that they were large enough to reach that less-than-whole-number-precision range. EDIT: For 32-bit floats, that's apparently any number above 16,777,216. Which seems surprisingly small to me. For 64-bit doubles that goes to 9,007,199,254,740,992. https://blog.demofox.org/2017/11/21/floating-point-precision https://blog.demofox.org/2017/11/21/floating-point-precision
  • jkaptur 4y ago
    I think the comment may have been edited ;) Anyway, yes, that's all true. Excel, Sheets, JavaScript numbers, Java doubles, Python floats, and so on all work this way. That's why I asked how switching spreadsheet implementations solved the problem.
    • jdmichal 4y ago
      Ah, sorry for that I must have seen the updated one. I agree with you that switching programs that are all using the IEEE floating-point formats with the same underlying hardware implementations should result in the same answers.