3 ms·

The OSM datamodel still revolves around a single datafile containing all data. At most it can be chopped up intogeographical bounded areas: the entire file for

by berkes 4y ago

The OSM datamodel still revolves around a single datafile containing all data. At most it can be chopped up intogeographical bounded areas: the entire file for a country, state, city or neighborhood. So, while I applaud and love micromapping (every door, bench, manhole or street lamp) it no longer scales. There are great apps to map every single tree, speed bump, lamp, bycicle stand, surface, roof and such. All great apps. All data accumulating in a still rapidly growing datafile. OSM really could use a form of layering. Especially when great apps like these here take off. So that users of the data can extract relevant data without having to parse giga, or terabytes of XML or PBFs.

pietervdvn 4y ago
No. OSM could _not_ have grown this big if we had layered everything. Far from it - the big power of OSM is the integration of all those 'layers'. Furhtermore, layering would needlessly complicate everything. Take a railway crossing for example. Should it go in the layer "railways", in the layer "roads" or in the layer "railway crossings"? Should we make a new layer for paths, so that they are separate from car-only roads? What if something changes? How should an object be moved from one layer to another? What with benches that double as a piece of artwork? What about this place that is a boardgame shop, a café and a social project for mentally disabled people? What if a river doubles as administrative boundary? If you want to extract data that is relevant for you, there is overpass-turbo.eu for this, where you can write a precise query and only get and download the data you need.
- berkes 4y ago
  I certainly agree it wouldn't have grown as big as it is with a less pragmatic approach. Entirely. But there must be a moment when some simplicity or pragmatism must be exchanged for scalability. Layering is one option. While indeed railroads and roads probably want to be on the same layer, opening hours or pavement need not be. I'm not saying a node may only ever be on one layer. They can easily be duplicated over layers. Just like at boundaries of geographical chunks, (i.e. over the borders of luxembourg.osm.pbf) there are (tens) of thousands of nodes that also appear in Germany, Belgium, the Netherlands once even.
  - pietervdvn 4y ago
    This 'splitting up in layers' gets done by third-parties, such as geofabrik. Geodata is more complicated then just 'layering' everything. Also, making an entire ecosystem change is a huge effort
    - berkes 4y ago
      I know and I agree. But "making an entire ecosystem change being a huge effort" does not prove that it's unnecessary. It just says that it is hard. I've never seen the OSM community back away from projects because they are "hard". And the moment this community starts doing that, it can just as well abolish oneself, IMO.
teraflop 4y ago
> The OSM datamodel still revolves around a single datafile containing all data. You could make an argument that this is not true: at an implementation level, the more fundamental OSM data model is a really big PostGIS database, which has the appropriate indexes to let you query whatever subsets you want. The big flat-file XML/PBF dumps are just one way that this data is exposed. You can also query the API for objects within a bounding box (and thereby benefit from the DB indexes), or you can use the daily/hourly/minutely incremental diffs to run your own database replica.
SnooSux 4y ago
> terabytes of XML or PBFs. I have a snapshot of all of OSM for a given date (conveniently called planet-XXXXXX.osm.pbf) and the compressed version is 62GB. And the format is optimized for reading certain chunks (as well as updating with new data). Plus there are libraries that do the heavy lifting for you in little time at all. The size is really not as big of an issue as you think it is.
- simlevesque 4y ago
  The planet file is currently 120GB for the plaintext. https://planet.osm.org/ https://planet.osm.org/
  - lukeqsee 4y ago
    No one uses the plain text, though. Every tool I have used, uses the PBF. (I run a company built on OSM, so that's a lot of tools.)
    - simlevesque 4y ago
      I know. I was just saying that because the person before said the plaintext was "terabytes".
- berkes 4y ago
  I use these tools. I've contributed to some. The size really is as big of an issue as I think it is

by berkes 4y ago

pietervdvn 4y ago

No. OSM could _not_ have grown this big if we had layered everything. Far from it - the big power of OSM is the integration of all those 'layers'. Furhtermore, layering would needlessly complicate everything. Take a railway crossing for example. Should it go in the layer "railways", in the layer "roads" or in the layer "railway crossings"? Should we make a new layer for paths, so that they are separate from car-only roads? What if something changes? How should an object be moved from one layer to another? What with benches that double as a piece of artwork? What about this place that is a boardgame shop, a café and a social project for mentally disabled people? What if a river doubles as administrative boundary? If you want to extract data that is relevant for you, there is overpass-turbo.eu for this, where you can write a precise query and only get and download the data you need.

berkes 4y ago
I certainly agree it wouldn't have grown as big as it is with a less pragmatic approach. Entirely. But there must be a moment when some simplicity or pragmatism must be exchanged for scalability. Layering is one option. While indeed railroads and roads probably want to be on the same layer, opening hours or pavement need not be. I'm not saying a node may only ever be on one layer. They can easily be duplicated over layers. Just like at boundaries of geographical chunks, (i.e. over the borders of luxembourg.osm.pbf) there are (tens) of thousands of nodes that also appear in Germany, Belgium, the Netherlands once even.
- pietervdvn 4y ago
  This 'splitting up in layers' gets done by third-parties, such as geofabrik. Geodata is more complicated then just 'layering' everything. Also, making an entire ecosystem change is a huge effort
  - berkes 4y ago
    I know and I agree. But "making an entire ecosystem change being a huge effort" does not prove that it's unnecessary. It just says that it is hard. I've never seen the OSM community back away from projects because they are "hard". And the moment this community starts doing that, it can just as well abolish oneself, IMO.

teraflop 4y ago

> The OSM datamodel still revolves around a single datafile containing all data. You could make an argument that this is not true: at an implementation level, the more fundamental OSM data model is a really big PostGIS database, which has the appropriate indexes to let you query whatever subsets you want. The big flat-file XML/PBF dumps are just one way that this data is exposed. You can also query the API for objects within a bounding box (and thereby benefit from the DB indexes), or you can use the daily/hourly/minutely incremental diffs to run your own database replica.

SnooSux 4y ago

> terabytes of XML or PBFs. I have a snapshot of all of OSM for a given date (conveniently called planet-XXXXXX.osm.pbf) and the compressed version is 62GB. And the format is optimized for reading certain chunks (as well as updating with new data). Plus there are libraries that do the heavy lifting for you in little time at all. The size is really not as big of an issue as you think it is.

simlevesque 4y ago
The planet file is currently 120GB for the plaintext. https://planet.osm.org/ https://planet.osm.org/
- lukeqsee 4y ago
  No one uses the plain text, though. Every tool I have used, uses the PBF. (I run a company built on OSM, so that's a lot of tools.)
  - simlevesque 4y ago
    I know. I was just saying that because the person before said the plaintext was "terabytes".
berkes 4y ago
I use these tools. I've contributed to some. The size really is as big of an issue as I think it is