4 ms·

I get why OSM does this, and it’s not necessarily a bad thing. The most important thing is being transparent about what format the data is in and always includi

I get why OSM does this, and it’s not necessarily a bad thing. The most important thing is being transparent about what format the data is in and always including that information in the metadata. the article touches on this point in this blurb: We should inform data consumers, so that they can decide to convert the coordinates to WGS84 if they want to. As someone who works in the GIS space, if we get data without this crucial information it’s sometimes very difficult to find the original datum to perform transformations and usually amounts to some form of educated guesswork based on the warp from WGS84 and the origin of the data “what datum/projection would I have used if I was the data creator?”

nwallin 7y ago
It really is a nightmare. It's like time zones, only when the datum is wrong, it's not necessarily immediately obvious what the correct one should be. People, even the ones who ought to know better, don't even seem to think it's necessary to specify their datum. It's bizarrely common to get a giant spreadsheet with coordinates in it and no datum. I've never seen data with its vertical datum specified. Don't get me started on the situation in China. https://en.wikipedia.org/wiki/Restrictions_on_geographic_data_in_China https://en.wikipedia.org/wiki/Restrictions_on_geographic_dat...
- scblock 7y ago
  Absolutely this. Or I'll get a CAD export in "Wyoming State Plane" but no indication that the designer decided to work in inches, not (US survey) feet. I spend more time than I ever would have guessed playing datum detective.
- incompatible 7y ago
  The problem is that the errors are too small for most users to care about. Few care if a date-time value from 2015 is a few hours out or if a location in Australia from 2008 is a metre out. So hardly anyone cares about finding a solution. The problem would be taken a lot more seriously if continental drift was 10m per year instead of a few cm.
ska 7y ago
This is a problem all over data processing ... once metadata becomes separate from the data over time it can become hard or impossible to correct later. I always push for detailed metadata, preferably in the same location/artifact. I'm regularly surprised by people who don't understand why it's important and just think "it's too much hassle".
- mulmen 7y ago
  For years we suffered at the hands of Oracle so developers refused to record time zones "for performance". This is no longer a technical concern but the cultural damage has been done.
- 7952 7y ago
  In GIS we have a system designed specifically for storing data which we tell people to use. But when it comes to metadata people use separate systems which need their own interface. And usually the metadata standard is far more complicated than the spatial data format! This seems crazy to me and just demonstrates how limited a lot of our technology is.
  - ska 7y ago
    I suspect this is common in lots of areas. At best you have a dedicated data catalog that is used consistently but still has a different interface. At worst you get a dogs breakfast.
  - Waterluvian 7y ago
    We also suffer from there being hundreds of formats for geodata and with that hundreds of ways Metadata is stored. And they don't always line up perfectly to allow seamless conversions.
- SOLAR_FIELDS 7y ago
  One thing that might be nice to do is to include an entirely WGS84 transformed version of the data for people looking to perform worldwide geospatial analysis. if the datum/projections are available for the regional datasets it’s a relatively trivial program to write as part of the ETL processing needed to make data available (PROJ implementations such as proj4j have everything you need to do this if the input metadata is available). Might need a decent amount of compute to perform all that transformation on a regular basis, but if it’s something like a weekly dump it doesn’t seem unrealistic at all. I think an optimized Spark job using proj4j on worldwide OSM data could probably perform this task in a few hours.
makomk 7y ago
The trouble is, the documentation is just wrong - all the official documentation says that OSM is in WGS-84. When I realised that this couldn't actually work due to plate tectonics and tried to figure out what they were actually doing, all I found was an April fool's joke from a couple of years ago: https://blog.openstreetmap.org/2017/03/31/osm-plate-tectonics/ https://blog.openstreetmap.org/2017/03/31/osm-plate-tectonic...
- not2b 7y ago
  I've contributed to OSM, fixing trail maps based on traces I recorded with my Garmin GPSMAP 64 which is configured to use WGS-84. If OSM wants something else they'd better let their contributors know how to deal with this.
  - makomk 7y ago
    It doesn't matter too much when using normal GPS receivers for mapping because the difference between datums is less than the position error. The main reason I was looking into it is because I was experimenting with a GPS setup that could theoretically achieve high enough accuracy for this to matter.
- maxerickson 7y ago
  Interpreting OSM as if it mattered would anyway be a mistake. The data does not have consistent or reliable accuracy and precision. Edit: Someone has downvoted this. It's an important point to remember when using OSM. Many GIS datasets are developed in a way that the accuracy and precision of the data is consistent across the entire dataset. OSM doesn't bother with this, because it's really hard to do, especially globally. That doesn't make OSM less useful for making consumer maps (there's no problem if data is off by 2 or 4 meters or whatever), but it can be a problem if say you were trying to make a legal determination about property ownership. OSM isn't a useful source for the latter task and would have to fundamentally change to become so.
  - mikekchar 7y ago
    To back this up, when I was making maps of the rural area where I live in Japan, there were a few places where aerial maps were obscured by trees and where GPS measurements were wildly inaccurate due to geographical features. So I just guessed where the road was. It's in there somewhere and if you are driving your car or riding your bike, it's more than good enough. If you needed to know the precise location of that road for another purpose? Totally useless. Maybe someone has gone in and improved my initial efforts, maybe not (although, I was looking at it the other day and I'm pretty sure they have).

nwallin 7y ago

It really is a nightmare. It's like time zones, only when the datum is wrong, it's not necessarily immediately obvious what the correct one should be. People, even the ones who ought to know better, don't even seem to think it's necessary to specify their datum. It's bizarrely common to get a giant spreadsheet with coordinates in it and no datum. I've never seen data with its vertical datum specified. Don't get me started on the situation in China. https://en.wikipedia.org/wiki/Restrictions_on_geographic_data_in_China https://en.wikipedia.org/wiki/Restrictions_on_geographic_dat...

scblock 7y ago
Absolutely this. Or I'll get a CAD export in "Wyoming State Plane" but no indication that the designer decided to work in inches, not (US survey) feet. I spend more time than I ever would have guessed playing datum detective.
incompatible 7y ago
The problem is that the errors are too small for most users to care about. Few care if a date-time value from 2015 is a few hours out or if a location in Australia from 2008 is a metre out. So hardly anyone cares about finding a solution. The problem would be taken a lot more seriously if continental drift was 10m per year instead of a few cm.

ska 7y ago

This is a problem all over data processing ... once metadata becomes separate from the data over time it can become hard or impossible to correct later. I always push for detailed metadata, preferably in the same location/artifact. I'm regularly surprised by people who don't understand why it's important and just think "it's too much hassle".

mulmen 7y ago
For years we suffered at the hands of Oracle so developers refused to record time zones "for performance". This is no longer a technical concern but the cultural damage has been done.
7952 7y ago
In GIS we have a system designed specifically for storing data which we tell people to use. But when it comes to metadata people use separate systems which need their own interface. And usually the metadata standard is far more complicated than the spatial data format! This seems crazy to me and just demonstrates how limited a lot of our technology is.
- ska 7y ago
  I suspect this is common in lots of areas. At best you have a dedicated data catalog that is used consistently but still has a different interface. At worst you get a dogs breakfast.
- Waterluvian 7y ago
  We also suffer from there being hundreds of formats for geodata and with that hundreds of ways Metadata is stored. And they don't always line up perfectly to allow seamless conversions.
SOLAR_FIELDS 7y ago
One thing that might be nice to do is to include an entirely WGS84 transformed version of the data for people looking to perform worldwide geospatial analysis. if the datum/projections are available for the regional datasets it’s a relatively trivial program to write as part of the ETL processing needed to make data available (PROJ implementations such as proj4j have everything you need to do this if the input metadata is available). Might need a decent amount of compute to perform all that transformation on a regular basis, but if it’s something like a weekly dump it doesn’t seem unrealistic at all. I think an optimized Spark job using proj4j on worldwide OSM data could probably perform this task in a few hours.

makomk 7y ago

The trouble is, the documentation is just wrong - all the official documentation says that OSM is in WGS-84. When I realised that this couldn't actually work due to plate tectonics and tried to figure out what they were actually doing, all I found was an April fool's joke from a couple of years ago: https://blog.openstreetmap.org/2017/03/31/osm-plate-tectonics/ https://blog.openstreetmap.org/2017/03/31/osm-plate-tectonic...

not2b 7y ago
I've contributed to OSM, fixing trail maps based on traces I recorded with my Garmin GPSMAP 64 which is configured to use WGS-84. If OSM wants something else they'd better let their contributors know how to deal with this.
- makomk 7y ago
  It doesn't matter too much when using normal GPS receivers for mapping because the difference between datums is less than the position error. The main reason I was looking into it is because I was experimenting with a GPS setup that could theoretically achieve high enough accuracy for this to matter.
maxerickson 7y ago
Interpreting OSM as if it mattered would anyway be a mistake. The data does not have consistent or reliable accuracy and precision. Edit: Someone has downvoted this. It's an important point to remember when using OSM. Many GIS datasets are developed in a way that the accuracy and precision of the data is consistent across the entire dataset. OSM doesn't bother with this, because it's really hard to do, especially globally. That doesn't make OSM less useful for making consumer maps (there's no problem if data is off by 2 or 4 meters or whatever), but it can be a problem if say you were trying to make a legal determination about property ownership. OSM isn't a useful source for the latter task and would have to fundamentally change to become so.
- mikekchar 7y ago
  To back this up, when I was making maps of the rural area where I live in Japan, there were a few places where aerial maps were obscured by trees and where GPS measurements were wildly inaccurate due to geographical features. So I just guessed where the road was. It's in there somewhere and if you are driving your car or riding your bike, it's more than good enough. If you needed to know the precise location of that road for another purpose? Totally useless. Maybe someone has gone in and improved my initial efforts, maybe not (although, I was looking at it the other day and I'm pretty sure they have).