4 ms·

When I was playing around with hosting rendered Minecraft slippy maps, I found the upload overhead to S3 to be pretty poor due to the “lots of tiny files” probl

When I was playing around with hosting rendered Minecraft slippy maps, I found the upload overhead to S3 to be pretty poor due to the “lots of tiny files” problem. (It renders as hundreds of thousands of rasterized image files for the various zoom levels). You could parallelize it, but it was still poor, and made me want to come up with a way to pack them into fewer files and fetch portions of them at render time so that the upload could be faster. It sounds like that’s what you did for the the PMTiles format?

bdon 6y ago
That's exactly what PMTiles is for, and there's nothing in it specific to maps or geographic coordinates, just a Z/X/Y pyramid structure. I chose to support Leaflet for now because it is very popular; there's other viewers like OpenSeadragon for which a decoder plugin would be possible.
- TheTaytay 6y ago
  Fantastic. The renderer I was using was indeed leaflet based, so this sounds drop-in easy. I just read through https://github.com/protomaps/PMTiles https://github.com/protomaps/PMTiles Nice work! The only downside would be that these maps are often re-rendered periodically and only have a few tiles that actually get updated. In that case, it’s nice to have to update only a handful of files. But I think the benefits would far outweigh the costs. Upload one massive file repeatedly is still gonna be faster I would think. Thanks!
incanus77 6y ago
That was the impetus behind MBTiles as well. I was building out an iOS toolkit to render OSM-based maps on iPad, offline, and transferring 1000s of rasters (at the time) even over USB was way too slow and error-prone.
- TheTaytay 6y ago
  Oh wow - very cool. Thanks for this! I love SQLite, and think this is an excellent use case. (File systems, especially networked ones, don’t handle “lots of tiny files” well). I think using SQLite as the primary, writeable storage mechanism makes sense because I could ship that around easily to rendering nodes and incrementally update it easily. Then, I could export to a static PMTile for upload to S3 for serving purposes.

bdon 6y ago

That's exactly what PMTiles is for, and there's nothing in it specific to maps or geographic coordinates, just a Z/X/Y pyramid structure. I chose to support Leaflet for now because it is very popular; there's other viewers like OpenSeadragon for which a decoder plugin would be possible.

TheTaytay 6y ago
Fantastic. The renderer I was using was indeed leaflet based, so this sounds drop-in easy. I just read through https://github.com/protomaps/PMTiles https://github.com/protomaps/PMTiles Nice work! The only downside would be that these maps are often re-rendered periodically and only have a few tiles that actually get updated. In that case, it’s nice to have to update only a handful of files. But I think the benefits would far outweigh the costs. Upload one massive file repeatedly is still gonna be faster I would think. Thanks!

incanus77 6y ago

That was the impetus behind MBTiles as well. I was building out an iOS toolkit to render OSM-based maps on iPad, offline, and transferring 1000s of rasters (at the time) even over USB was way too slow and error-prone.

TheTaytay 6y ago
Oh wow - very cool. Thanks for this! I love SQLite, and think this is an excellent use case. (File systems, especially networked ones, don’t handle “lots of tiny files” well). I think using SQLite as the primary, writeable storage mechanism makes sense because I could ship that around easily to rendering nodes and incrementally update it easily. Then, I could export to a static PMTile for upload to S3 for serving purposes.