3 ms·

New OSM file format: 30% smaller than PBF, 5x faster to import

145 points by raybb 1y ago

shoo 1y ago
I was curious about the spec of this new GOB format, there's a comment down thread explaining that there isn't a spec yet but discussing some of the details of the format: https://community.openstreetmap.org/t/new-osm-file-format-30-smaller-than-pbf-5x-faster-to-import/137151/9 https://community.openstreetmap.org/t/new-osm-file-format-30... Aside from OSM specifics, performance friendly formats for spatial data that support spatial indexing can make huge impact on usability and productivity of applications. e.g. trying to view a large dataset in QGIS that has been saved as KMZ (zipped XML) can make QGIS basically hang for minutes, while the same dataset saved as something like flatgeobuf [1] can be loaded instantly. [1] https://flatgeobuf.org/ https://flatgeobuf.org/
- rtpg 1y ago
  Sometimes with QGIS the best thing you can do is load up the stuff into Postgres, just orders of magnitude of perf improvements
- brailsafe 1y ago
  My guess is that one of the fundamental differences there would be that KMZ isn't streamable and needs to be fully loaded into memory and then transformed into whatever structure qgis uses internally, but I'm not totally sure about that and haven't used QGIS in a minute. I feel like I've also had bad luck loading KMZ/KML of any reasonable complexity into any other GIS app. How's geojson of the same data?
tobwen 1y ago
My opinion: Without support in libosmium and GDAL, this will remain a marginal phenomenon.
- Groxx 1y ago
  is there some reason to believe they will not support it? because otherwise this is true of all new ̶s̶p̶e̶c̶s̶ edit: ideas (this isn't even a finished spec), and it implies absolutely nothing.
PLenz 1y ago
Does it work with osmium?
- odo1242 1y ago
  Not yet - it was just introduced and there isn’t even a full spec yet
thot_experiment 1y ago
Tangentially related question for any of you GIS people who might be lurking in this thread: Can anyone recommend me a method of meshing LIDAR point clouds? The sparseness of the data on building walls & other near-vertical surfaces combined with a lack of point normals leads to degenerate solutions with all the common approaches (poisson/ball pivot/vcg in meshlab) not to mention extremely slow perf. Tree canopies and overhanging parapets make a simple heightmap approach less-than desirable (though ultimately acceptable if I can't find anything better). I'm trying to turn 90 billion lidar points into maybe 30-50 million triangles, hopefully without spending months developing a custom pipeline.
- shrinks99 1y ago
  I think Meshroom can use LIDAR data as an input now? I used it years ago for photogrammetry and camera tracking for some VFX work and it's an incredibly solid suite of open source tools for these types of tasks.
- denpa 1y ago
  https://3dbag.nl/ https://3dbag.nl/ might be worth a try. This project reconstructed and maintains building models of 11 million buildings in the Netherlands. It combines airborne LiDAR and building footprints, it's OS (https://github.com/3DBAG https://github.com/3DBAG) with the reconstruction pipeline here: https://github.com/3DBAG/roofer https://github.com/3DBAG/roofer.
zigzag312 1y ago
Does this use the new OSM data model? https://media.jochentopf.com/media/2022-08-15-study-evolution-of-the-osm-data-model.pdf https://media.jochentopf.com/media/2022-08-15-study-evolutio... https://github.com/osmlab/osm-data-model https://github.com/osmlab/osm-data-model https://blog.openstreetmap.org/2023/01/04/reminder-call-for-feedback-on-the-data-model/ https://blog.openstreetmap.org/2023/01/04/reminder-call-for-... Resolving the coordinates to node references in current data model is such a nuisance as it's slow and requires lots of RAM.
ramanvarma 1y ago
does the tiling approach have any trade-offs on random access or is lookup performance comparable to PBF once loaded?

145 points by raybb 1y ago

shoo 1y ago

I was curious about the spec of this new GOB format, there's a comment down thread explaining that there isn't a spec yet but discussing some of the details of the format: https://community.openstreetmap.org/t/new-osm-file-format-30-smaller-than-pbf-5x-faster-to-import/137151/9 https://community.openstreetmap.org/t/new-osm-file-format-30... Aside from OSM specifics, performance friendly formats for spatial data that support spatial indexing can make huge impact on usability and productivity of applications. e.g. trying to view a large dataset in QGIS that has been saved as KMZ (zipped XML) can make QGIS basically hang for minutes, while the same dataset saved as something like flatgeobuf [1] can be loaded instantly. [1] https://flatgeobuf.org/ https://flatgeobuf.org/

rtpg 1y ago
Sometimes with QGIS the best thing you can do is load up the stuff into Postgres, just orders of magnitude of perf improvements
brailsafe 1y ago
My guess is that one of the fundamental differences there would be that KMZ isn't streamable and needs to be fully loaded into memory and then transformed into whatever structure qgis uses internally, but I'm not totally sure about that and haven't used QGIS in a minute. I feel like I've also had bad luck loading KMZ/KML of any reasonable complexity into any other GIS app. How's geojson of the same data?

tobwen 1y ago

My opinion: Without support in libosmium and GDAL, this will remain a marginal phenomenon.

Groxx 1y ago
is there some reason to believe they will not support it? because otherwise this is true of all new ̶s̶p̶e̶c̶s̶ edit: ideas (this isn't even a finished spec), and it implies absolutely nothing.

PLenz 1y ago

Does it work with osmium?

odo1242 1y ago
Not yet - it was just introduced and there isn’t even a full spec yet

thot_experiment 1y ago

Tangentially related question for any of you GIS people who might be lurking in this thread: Can anyone recommend me a method of meshing LIDAR point clouds? The sparseness of the data on building walls & other near-vertical surfaces combined with a lack of point normals leads to degenerate solutions with all the common approaches (poisson/ball pivot/vcg in meshlab) not to mention extremely slow perf. Tree canopies and overhanging parapets make a simple heightmap approach less-than desirable (though ultimately acceptable if I can't find anything better). I'm trying to turn 90 billion lidar points into maybe 30-50 million triangles, hopefully without spending months developing a custom pipeline.

shrinks99 1y ago
I think Meshroom can use LIDAR data as an input now? I used it years ago for photogrammetry and camera tracking for some VFX work and it's an incredibly solid suite of open source tools for these types of tasks.
denpa 1y ago
https://3dbag.nl/ https://3dbag.nl/ might be worth a try. This project reconstructed and maintains building models of 11 million buildings in the Netherlands. It combines airborne LiDAR and building footprints, it's OS (https://github.com/3DBAG https://github.com/3DBAG) with the reconstruction pipeline here: https://github.com/3DBAG/roofer https://github.com/3DBAG/roofer.

zigzag312 1y ago

Does this use the new OSM data model? https://media.jochentopf.com/media/2022-08-15-study-evolution-of-the-osm-data-model.pdf https://media.jochentopf.com/media/2022-08-15-study-evolutio... https://github.com/osmlab/osm-data-model https://github.com/osmlab/osm-data-model https://blog.openstreetmap.org/2023/01/04/reminder-call-for-feedback-on-the-data-model/ https://blog.openstreetmap.org/2023/01/04/reminder-call-for-... Resolving the coordinates to node references in current data model is such a nuisance as it's slow and requires lots of RAM.

ramanvarma 1y ago

does the tiling approach have any trade-offs on random access or is lookup performance comparable to PBF once loaded?