3 ms·
> Using 16-bit floats for lat/lon gives you accuracy down to 1 meter. Not for Longitude it doesn't with values > abs(128), as that for example means 132.0 has
by pixelesque 1y ago
> Using 16-bit floats for lat/lon gives you accuracy down to 1 meter.
Not for Longitude it doesn't with values > abs(128), as that for example means 132.0 has the next possible value of 132.125.
float16 precision at values > 16 is pretty poor.
Converting that discrepency (132.125 - 132.0) to KM gives 10 KM.
Did you maybe mean Fixed-point? (but even then that's not enough precision for 1m)
- tantalor 1y agoGood catch, I didn't consider that.