3 ms·

This is a problem all over data processing ... once metadata becomes separate from the data over time it can become hard or impossible to correct later. I alwa

by ska 7y ago

This is a problem all over data processing ... once metadata becomes separate from the data over time it can become hard or impossible to correct later. I always push for detailed metadata, preferably in the same location/artifact. I'm regularly surprised by people who don't understand why it's important and just think "it's too much hassle".

mulmen 7y ago
For years we suffered at the hands of Oracle so developers refused to record time zones "for performance". This is no longer a technical concern but the cultural damage has been done.
7952 7y ago
In GIS we have a system designed specifically for storing data which we tell people to use. But when it comes to metadata people use separate systems which need their own interface. And usually the metadata standard is far more complicated than the spatial data format! This seems crazy to me and just demonstrates how limited a lot of our technology is.
- ska 7y ago
  I suspect this is common in lots of areas. At best you have a dedicated data catalog that is used consistently but still has a different interface. At worst you get a dogs breakfast.
- Waterluvian 7y ago
  We also suffer from there being hundreds of formats for geodata and with that hundreds of ways Metadata is stored. And they don't always line up perfectly to allow seamless conversions.
SOLAR_FIELDS 7y ago
One thing that might be nice to do is to include an entirely WGS84 transformed version of the data for people looking to perform worldwide geospatial analysis. if the datum/projections are available for the regional datasets it’s a relatively trivial program to write as part of the ETL processing needed to make data available (PROJ implementations such as proj4j have everything you need to do this if the input metadata is available). Might need a decent amount of compute to perform all that transformation on a regular basis, but if it’s something like a weekly dump it doesn’t seem unrealistic at all. I think an optimized Spark job using proj4j on worldwide OSM data could probably perform this task in a few hours.

by ska 7y ago

mulmen 7y ago

For years we suffered at the hands of Oracle so developers refused to record time zones "for performance". This is no longer a technical concern but the cultural damage has been done.

7952 7y ago

In GIS we have a system designed specifically for storing data which we tell people to use. But when it comes to metadata people use separate systems which need their own interface. And usually the metadata standard is far more complicated than the spatial data format! This seems crazy to me and just demonstrates how limited a lot of our technology is.

ska 7y ago
I suspect this is common in lots of areas. At best you have a dedicated data catalog that is used consistently but still has a different interface. At worst you get a dogs breakfast.
Waterluvian 7y ago
We also suffer from there being hundreds of formats for geodata and with that hundreds of ways Metadata is stored. And they don't always line up perfectly to allow seamless conversions.

SOLAR_FIELDS 7y ago

One thing that might be nice to do is to include an entirely WGS84 transformed version of the data for people looking to perform worldwide geospatial analysis. if the datum/projections are available for the regional datasets it’s a relatively trivial program to write as part of the ETL processing needed to make data available (PROJ implementations such as proj4j have everything you need to do this if the input metadata is available). Might need a decent amount of compute to perform all that transformation on a regular basis, but if it’s something like a weekly dump it doesn’t seem unrealistic at all. I think an optimized Spark job using proj4j on worldwide OSM data could probably perform this task in a few hours.