3 ms·

You thought OpenStreetMap uses WGS84? No it doesn't (2019)

109 points by fanf2 2y ago

gen3 2y ago
Good post, I assume this was in response to the "Every map of China is wrong" post a few weeks ago? https://news.ycombinator.com/item?id=40220072 https://news.ycombinator.com/item?id=40220072 Its not surprising to me that there are so many competing standards, I'm curious how that ends up being rectified in the software (is it all internally translated on ingest?)
- michaelt 2y ago
  The standards mentioned in the article aren't really competing standards in the normal sense. It's simply that the continents drift relative to one another. So if you want to make a map of Australia and the continent is drifting by 3 inches per year relative to other continents, you often want to define your map's coordinates as relative to some fixed point on your continent.
  - lxgr 2y ago
    Oh, the luxury of having a continent-sized country entirely located on (and far from the borders of) a single tectonic plate :)
    - johannes1234321 2y ago
      Sad if such a country moves about 1.5m in just 20 years and thus offsets any external reference (including satellites for GPS and imaging) https://www.sciencealert.com/australia-s-about-to-move-1-5-metres-to-the-north https://www.sciencealert.com/australia-s-about-to-move-1-5-m...
      - lxgr 2y ago
        That's true for all countries, since every single one is located on a moving tectonic plate! But some countries experience little "intra-country coordinate drift" per year, which can make local/regional geodetic datums attractive. That approach wouldn't work for the US, for example: https://en.wikipedia.org/wiki/San_Andreas_Fault https://en.wikipedia.org/wiki/San_Andreas_Fault
- solardev 2y ago
  This post was from 2019, and is standard mapping stuff not necessarily having to do with any Chinese obfuscation. It's not necessarily a case of "competing" standards, but choosing to apply local optimizations (or not) for better precision, at a cost of increased error (imprecision) elsewhere. The Earth isn't a perfect static sphere, it's a roundish blob with dynamic bumps that crawl and ooze along its surface over time. You can kinda try to average it out everywhere (WGS84) or if your area of concern is smaller, choose a datum that's more locally accurate but that would be inappropriate if used elsewhere. The Wiki article is a better explanation I think: https://en.wikipedia.org/wiki/Geodetic_datum?wprov=sfla1 https://en.wikipedia.org/wiki/Geodetic_datum?wprov=sfla1 The hard part for OSM (and any long lived community effort) is that their data sources are varied, and they have to first identify the source datum (sometimes it's in metadata but not always, and might need to be manually measured) and then decide what to do about it, on a case by case basis. In the article it didn't seem like they'd made a decision yet. Wikipedia talks more about the "rectification" (datum transformation) process more: https://en.wikipedia.org/wiki/Geographic_coordinate_conversion#Datum_transformations?wprov=sfla1 https://en.wikipedia.org/wiki/Geographic_coordinate_conversi... You can download the FOSS app QGIS and play around with its transformation and reprojection tools if you want to see how software handles this. The algorithms it uses under the hood are also open source CLI tools so you can check out the math there if you're interested. (I love maps but hate math, so that part is all black box magic to me lol).
  - ryandrake 2y ago
    Right. The way I was taught is that a latitude/longitude cannot be accurately plotted on a map unless you know the lat/lon's datum and the map's datum, and ensure they match. Naive applications might assume a popular datum like WGS84, but not every map uses WGS84, so your locations will be plotted incorrectly if there is a mismatch. Kind of like how many developers now just assume a string of bytes called "text" is encoded in UTF-8, but if they are actually encoded differently, you may not render the string correctly. Ideally, every latitude/longitude pair entered into OSM should specify datum as metadata, but as you say, the existing data is probably very incomplete.
    - 2y ago
ks2048 2y ago
There's some things here I don't understand well enough... but is there a difference between saying "OSM doesn't use WGS84" and "OSM uses WGS84, but some of the data is not imported accurately"? (apart the issue of plate movements over time).
- mtmail 2y ago
  I think you're correct. The post is 5 years old and data users and mappers (people editing the map) still use it the same: assume WGS84, be aware that sometimes in some regions that might not be 100% accurate. Those needing centimeter accuracy will use government data, for Australia that would be GDA2020 (which replaced GDA94). Car navigation (assuming a human driver) hardly needs meter accuracy. https://www.icsm.gov.au/gda2020 https://www.icsm.gov.au/gda2020
- deleted 2y ago
  [deleted]
binarymax 2y ago
Just here to say that open tech and data like OSM, Nominatim, PostGIS, and Leaflet are some of the most amazing tools out there, and they make extremely complicated things like this really easy to use. Serious respect and props to everyone contributing to these tools.
- solardev 2y ago
  Just want to put a shout out to OpenLayers and QGIS too! QGIS is ESRI's open source competitor (and IMO much easier to use). OpenLayers is like Leaflet but more powerful in some ways, especially when it comes to projections and transformations.
deleted 2y ago
[deleted]
lolc 2y ago
Oh wow, reality comes knocking just when I stopped laughing about China and their silly distorted reference system! https://news.ycombinator.com/item?id=40220072 https://news.ycombinator.com/item?id=40220072 From my armchair I recommend an automated edit at unprecedented scale: Apply tectonic shift to all nodes every year. This would allow keeping WGS84 while keeping the map closer to the present.
- solardev 2y ago
  This already happens! There's a global effort to measure and update these every few years, and that creates "realizations" (versions) of the ITRS reference frame: https://www.e-education.psu.edu/geog862/node/1803 https://www.e-education.psu.edu/geog862/node/1803
rich_sasha 2y ago
Most accurate military weaponry aimed at coordinates in space / on the ground (as opposed to at an object eg. optically) has accuracy around 3m, from memory (as far as these things are published). I wonder if eventually the continental drift becomes meaningful to that. From memory GPS, and so maybe also US military, use WGS84. I guess maybe not. Whatever aims the rocket is subject to the same drift, if the measurement and the firing are contemporaneous then the drift doesn't matter. Still, it's funny to imagine an angry general being told his missile missed because of continental drift. I once heard an apocryphal story about US generals being told GPS needs relativistic corrections to the clocks, and finding it ridiculous. For a few initial weeks, the correction was thus disabled. Only when they saw the measurements drift predictably by multiple meters per day did they relent.
- solardev 2y ago
  Do they ever aim anything at just a coordinate and let loose? For guided weapons, I thought there's usually a variety of sensors on board (GPS, inertial nav, terrain following, and then of course radar or IR seekers or some sort of visual camera)? Edit: Self-correction: Yes, there are. Common JDAM munitions use GPS and inertial guidance: https://en.wikipedia.org/wiki/Joint_Direct_Attack_Munition?wprov=sfla1 https://en.wikipedia.org/wiki/Joint_Direct_Attack_Munition?w...

109 points by fanf2 2y ago

gen3 2y ago

Good post, I assume this was in response to the "Every map of China is wrong" post a few weeks ago? https://news.ycombinator.com/item?id=40220072 https://news.ycombinator.com/item?id=40220072 Its not surprising to me that there are so many competing standards, I'm curious how that ends up being rectified in the software (is it all internally translated on ingest?)

michaelt 2y ago
The standards mentioned in the article aren't really competing standards in the normal sense. It's simply that the continents drift relative to one another. So if you want to make a map of Australia and the continent is drifting by 3 inches per year relative to other continents, you often want to define your map's coordinates as relative to some fixed point on your continent.
- lxgr 2y ago
  Oh, the luxury of having a continent-sized country entirely located on (and far from the borders of) a single tectonic plate :)
  - johannes1234321 2y ago
    Sad if such a country moves about 1.5m in just 20 years and thus offsets any external reference (including satellites for GPS and imaging) https://www.sciencealert.com/australia-s-about-to-move-1-5-metres-to-the-north https://www.sciencealert.com/australia-s-about-to-move-1-5-m...
    - lxgr 2y ago
      That's true for all countries, since every single one is located on a moving tectonic plate! But some countries experience little "intra-country coordinate drift" per year, which can make local/regional geodetic datums attractive. That approach wouldn't work for the US, for example: https://en.wikipedia.org/wiki/San_Andreas_Fault https://en.wikipedia.org/wiki/San_Andreas_Fault
solardev 2y ago
This post was from 2019, and is standard mapping stuff not necessarily having to do with any Chinese obfuscation. It's not necessarily a case of "competing" standards, but choosing to apply local optimizations (or not) for better precision, at a cost of increased error (imprecision) elsewhere. The Earth isn't a perfect static sphere, it's a roundish blob with dynamic bumps that crawl and ooze along its surface over time. You can kinda try to average it out everywhere (WGS84) or if your area of concern is smaller, choose a datum that's more locally accurate but that would be inappropriate if used elsewhere. The Wiki article is a better explanation I think: https://en.wikipedia.org/wiki/Geodetic_datum?wprov=sfla1 https://en.wikipedia.org/wiki/Geodetic_datum?wprov=sfla1 The hard part for OSM (and any long lived community effort) is that their data sources are varied, and they have to first identify the source datum (sometimes it's in metadata but not always, and might need to be manually measured) and then decide what to do about it, on a case by case basis. In the article it didn't seem like they'd made a decision yet. Wikipedia talks more about the "rectification" (datum transformation) process more: https://en.wikipedia.org/wiki/Geographic_coordinate_conversion#Datum_transformations?wprov=sfla1 https://en.wikipedia.org/wiki/Geographic_coordinate_conversi... You can download the FOSS app QGIS and play around with its transformation and reprojection tools if you want to see how software handles this. The algorithms it uses under the hood are also open source CLI tools so you can check out the math there if you're interested. (I love maps but hate math, so that part is all black box magic to me lol).
- ryandrake 2y ago
  Right. The way I was taught is that a latitude/longitude cannot be accurately plotted on a map unless you know the lat/lon's datum and the map's datum, and ensure they match. Naive applications might assume a popular datum like WGS84, but not every map uses WGS84, so your locations will be plotted incorrectly if there is a mismatch. Kind of like how many developers now just assume a string of bytes called "text" is encoded in UTF-8, but if they are actually encoded differently, you may not render the string correctly. Ideally, every latitude/longitude pair entered into OSM should specify datum as metadata, but as you say, the existing data is probably very incomplete.
  - 2y ago

ks2048 2y ago

There's some things here I don't understand well enough... but is there a difference between saying "OSM doesn't use WGS84" and "OSM uses WGS84, but some of the data is not imported accurately"? (apart the issue of plate movements over time).

mtmail 2y ago
I think you're correct. The post is 5 years old and data users and mappers (people editing the map) still use it the same: assume WGS84, be aware that sometimes in some regions that might not be 100% accurate. Those needing centimeter accuracy will use government data, for Australia that would be GDA2020 (which replaced GDA94). Car navigation (assuming a human driver) hardly needs meter accuracy. https://www.icsm.gov.au/gda2020 https://www.icsm.gov.au/gda2020
deleted 2y ago
[deleted]

binarymax 2y ago

Just here to say that open tech and data like OSM, Nominatim, PostGIS, and Leaflet are some of the most amazing tools out there, and they make extremely complicated things like this really easy to use. Serious respect and props to everyone contributing to these tools.

solardev 2y ago
Just want to put a shout out to OpenLayers and QGIS too! QGIS is ESRI's open source competitor (and IMO much easier to use). OpenLayers is like Leaflet but more powerful in some ways, especially when it comes to projections and transformations.

deleted 2y ago

[deleted]

lolc 2y ago

Oh wow, reality comes knocking just when I stopped laughing about China and their silly distorted reference system! https://news.ycombinator.com/item?id=40220072 https://news.ycombinator.com/item?id=40220072 From my armchair I recommend an automated edit at unprecedented scale: Apply tectonic shift to all nodes every year. This would allow keeping WGS84 while keeping the map closer to the present.

solardev 2y ago
This already happens! There's a global effort to measure and update these every few years, and that creates "realizations" (versions) of the ITRS reference frame: https://www.e-education.psu.edu/geog862/node/1803 https://www.e-education.psu.edu/geog862/node/1803

rich_sasha 2y ago

Most accurate military weaponry aimed at coordinates in space / on the ground (as opposed to at an object eg. optically) has accuracy around 3m, from memory (as far as these things are published). I wonder if eventually the continental drift becomes meaningful to that. From memory GPS, and so maybe also US military, use WGS84. I guess maybe not. Whatever aims the rocket is subject to the same drift, if the measurement and the firing are contemporaneous then the drift doesn't matter. Still, it's funny to imagine an angry general being told his missile missed because of continental drift. I once heard an apocryphal story about US generals being told GPS needs relativistic corrections to the clocks, and finding it ridiculous. For a few initial weeks, the correction was thus disabled. Only when they saw the measurements drift predictably by multiple meters per day did they relent.

solardev 2y ago
Do they ever aim anything at just a coordinate and let loose? For guided weapons, I thought there's usually a variety of sensors on board (GPS, inertial nav, terrain following, and then of course radar or IR seekers or some sort of visual camera)? Edit: Self-correction: Yes, there are. Common JDAM munitions use GPS and inertial guidance: https://en.wikipedia.org/wiki/Joint_Direct_Attack_Munition?wprov=sfla1 https://en.wikipedia.org/wiki/Joint_Direct_Attack_Munition?w...