3 ms·

Originally, MBTiles was designed for mobile offline. Then, Mapbox started serving directly from the SQLite for web & other clients, creating the Node.js SQLite

Originally, MBTiles was designed for mobile offline. Then, Mapbox started serving directly from the SQLite for web & other clients, creating the Node.js SQLite bindings in the process. Eventually, they had an unpacker for MBTiles uploads that would stick the tiles in CDN. So, really no more complex than old school static content hosting. Source: I created MBTiles ;-)

bdon 6y ago
Author of OP here, I use the tile unpacker as well! I didn't point this out in the OP, but I found the ergonomics of uploading tiles to S3 etc get bad once you're over a couple tens of thousands of tiles, as each upload has some overhead. I've tested PMTiles on S3 successfully with hundreds of millions of tiles. Deduplication is also important to save space on redundant ocean/earth tiles; MBTiles can deduplicate via views, PMTiles deduplicates by reference, but unpacked Z/X/Y tiles on S3 can't.
- incanus77 6y ago
  Neat! I'm hoping to dig into your design more... looks great.
- TheTaytay 6y ago
  When I was playing around with hosting rendered Minecraft slippy maps, I found the upload overhead to S3 to be pretty poor due to the “lots of tiny files” problem. (It renders as hundreds of thousands of rasterized image files for the various zoom levels). You could parallelize it, but it was still poor, and made me want to come up with a way to pack them into fewer files and fetch portions of them at render time so that the upload could be faster. It sounds like that’s what you did for the the PMTiles format?
  - bdon 6y ago
    That's exactly what PMTiles is for, and there's nothing in it specific to maps or geographic coordinates, just a Z/X/Y pyramid structure. I chose to support Leaflet for now because it is very popular; there's other viewers like OpenSeadragon for which a decoder plugin would be possible.
    - TheTaytay 6y ago
      Fantastic. The renderer I was using was indeed leaflet based, so this sounds drop-in easy. I just read through https://github.com/protomaps/PMTiles https://github.com/protomaps/PMTiles Nice work! The only downside would be that these maps are often re-rendered periodically and only have a few tiles that actually get updated. In that case, it’s nice to have to update only a handful of files. But I think the benefits would far outweigh the costs. Upload one massive file repeatedly is still gonna be faster I would think. Thanks!
  - incanus77 6y ago
    That was the impetus behind MBTiles as well. I was building out an iOS toolkit to render OSM-based maps on iPad, offline, and transferring 1000s of rasters (at the time) even over USB was way too slow and error-prone.
    - TheTaytay 6y ago
      Oh wow - very cool. Thanks for this! I love SQLite, and think this is an excellent use case. (File systems, especially networked ones, don’t handle “lots of tiny files” well). I think using SQLite as the primary, writeable storage mechanism makes sense because I could ship that around easily to rendering nodes and incrementally update it easily. Then, I could export to a static PMTile for upload to S3 for serving purposes.

bdon 6y ago

Author of OP here, I use the tile unpacker as well! I didn't point this out in the OP, but I found the ergonomics of uploading tiles to S3 etc get bad once you're over a couple tens of thousands of tiles, as each upload has some overhead. I've tested PMTiles on S3 successfully with hundreds of millions of tiles. Deduplication is also important to save space on redundant ocean/earth tiles; MBTiles can deduplicate via views, PMTiles deduplicates by reference, but unpacked Z/X/Y tiles on S3 can't.

incanus77 6y ago
Neat! I'm hoping to dig into your design more... looks great.
TheTaytay 6y ago
When I was playing around with hosting rendered Minecraft slippy maps, I found the upload overhead to S3 to be pretty poor due to the “lots of tiny files” problem. (It renders as hundreds of thousands of rasterized image files for the various zoom levels). You could parallelize it, but it was still poor, and made me want to come up with a way to pack them into fewer files and fetch portions of them at render time so that the upload could be faster. It sounds like that’s what you did for the the PMTiles format?
- bdon 6y ago
  That's exactly what PMTiles is for, and there's nothing in it specific to maps or geographic coordinates, just a Z/X/Y pyramid structure. I chose to support Leaflet for now because it is very popular; there's other viewers like OpenSeadragon for which a decoder plugin would be possible.
  - TheTaytay 6y ago
    Fantastic. The renderer I was using was indeed leaflet based, so this sounds drop-in easy. I just read through https://github.com/protomaps/PMTiles https://github.com/protomaps/PMTiles Nice work! The only downside would be that these maps are often re-rendered periodically and only have a few tiles that actually get updated. In that case, it’s nice to have to update only a handful of files. But I think the benefits would far outweigh the costs. Upload one massive file repeatedly is still gonna be faster I would think. Thanks!
- incanus77 6y ago
  That was the impetus behind MBTiles as well. I was building out an iOS toolkit to render OSM-based maps on iPad, offline, and transferring 1000s of rasters (at the time) even over USB was way too slow and error-prone.
  - TheTaytay 6y ago
    Oh wow - very cool. Thanks for this! I love SQLite, and think this is an excellent use case. (File systems, especially networked ones, don’t handle “lots of tiny files” well). I think using SQLite as the primary, writeable storage mechanism makes sense because I could ship that around easily to rendering nodes and incrementally update it easily. Then, I could export to a static PMTile for upload to S3 for serving purposes.