3 ms·
> 0 correct digits when the output is close to 0 this is an amusing way to describe the precision of sub-normal floating point numbers
by enriquto 5y ago
> 0 correct digits when the output is close to 0
this is an amusing way to describe the precision of sub-normal floating point numbers
- lifthrasiir 5y agoIt is much more amusing if you describe it in ulps; for some inputs the error can reach > 2^90 ulps, more than the mantissa size itself.
- adgjlsfhk1 5y agoIt's not just sub-normal numbers. As https://randomascii.wordpress.com/2014/10/09/intel-underestimates-error-bounds-by-1-3-quintillion/ https://randomascii.wordpress.com/2014/10/09/intel-underesti... shows, fsin only uses 66 bits of pi, which means you have roughly no precision whenever abs(sin(x))<10^-16 which is way bigger than the biggest subnormal (10^-307 or so)
- ramshorns 5y agoIn that range, just returning x would be way better. Maybe even perfect actually - if x is less than 10^-16, then the error of x^3/6 is less than the machine precision for x.
- adgjlsfhk1 5y agothe error isn't when x is small. it's when sin(x) is small. the problem happens for x near multiples of pi