3 ms·
I wonder if they're using a fraction type to store decimals (a reasonable way to avoid floating-point error), and when the 10^x function is implemented naively
by rspeer 8y ago
I wonder if they're using a fraction type to store decimals (a reasonable way to avoid floating-point error), and when the 10^x function is implemented naively on this type, 10^(3.1415) becomes 10^(31415/10000) = (10^31415 / 10^10000) = (overflow / overflow).
- istjohn 8y ago> 10^(31415/10000) = (10^31415 / 10^10000) If n^(a/b) = (n^a)/(n^b) then 2^(1/2) = (2^1)/(2^2) = 2/4 = 0.5 but 2^(1/2) = sqrt(2) = 1.4...
- rspeer 8y agoDang, you're right, I should have known that. I apologize for my bad math. It would be the 10000th root of 10^31415, and at this point what I'm saying is probably entirely unrelated to how it's really implemented.