6 ms·

If anyone else was curious, their sample OSM derived PMTile file (protomaps-basemap-opensource-20230408.pmtiles) is 103GB. https://docs.protomaps.com/basemaps/

If anyone else was curious, their sample OSM derived PMTile file (protomaps-basemap-opensource-20230408.pmtiles) is 103GB. https://docs.protomaps.com/basemaps/downloads https://docs.protomaps.com/basemaps/downloads

pmontra 3y ago
It's 107 GB now. It would be great to be able to download only diffs and apply them to a daily, weekly, monthly baseline.
- blutack 3y ago
  Hmm I was just going off the Firefox estimate, maybe there's some browser difference?
  - bdon 3y ago
    The newest major version has moved to a daily build channel (the last major version was last updated in April) Today 10/23 is 107GB. https://maps.protomaps.com/builds/ https://maps.protomaps.com/builds/ I'll update the documentation page today.
    - bdon 3y ago
      (I've updated the page at https://docs.protomaps.com/basemaps/downloads https://docs.protomaps.com/basemaps/downloads)
- mycall 3y ago
  I wonder if the compressed Hilbert ordering inside the tiles make them hard to diff efficiently.
  - bdon 3y ago
    The ordering applies across tiles, and not within - it should be efficient to diff two archives because there is a single, well-defined ordering over the entire archive, and writing algorithms over the tileset is all 1D operations instead of 2D (a single zoom level plane) or 3D (multiple planes)
dinkblam 3y ago
thanks for the info, weird that this isn't answered in the FAQ
edhelas 3y ago
Exactly the kind of thing that could be downloaded using Bittorrent!
- ElectricalUnion 3y ago
  I assume that would require BitTorrent v2 (and I think more specifically something along BEP-46 - Updating Torrents Via DHT Mutable Items) or your map is gonna be out-of-sync and sad after a while.
  - nextaccountic 3y ago
    are the torrents in the wild that uses this? and are there bittorrent clients that implement this? how do they surface the mutable aspect to the user? (is it like, your torrent client will silently update files, and if you dont want to update anymore you need to pause? can you still seed if you dont want to update?)
    - anacrolix 3y ago
      I use it in anacrolix/btlink. There's support in anacrolix/dht.
      - nextaccountic 3y ago
        That's very cool! Does this serve decentralized "web app" through bittorrent? Do I need your client to seed it, or can something like qbittorrent seed it? Will I seed only the last version of the page or all versions? Can someone download an earlier version or list all past versions?
    - acalcutt 3y ago
      I've been testing a setup that automatically generates OpenStreetMap/OpenMapTiles formatted pmtiles and mbtiles, then makes them available by torrent. The torrents get placed in a rss feed at https://planetgen.wifidb.net/ https://planetgen.wifidb.net/ Basically what I did was set up 'qbittorent' client to watch the openstreetmap rss weekly torrent feed. when it downloads a new pbf file, it runs this script I made at https://github.com/acalcutt/PlanetilerTorrent https://github.com/acalcutt/PlanetilerTorrent , which was based of the osm torrent creation process. The script creates pmtiles and mbtiles using planetiler, then makes torrent and starts seeding them in qbittorent. In qbittorent I have options like how long I want to seed for (30 days), what do do when done seeding (Delete the files) and also speed controls so I can limit bandwidth during my working hours. I have this running on a old laptop with 2TB nvme/64GB memory. It seeems to work pretty well so far. It would be nice if my internet speeds were a little better for initial seeding, but at least the torrents share the load with other people who are downloading/
bambax 3y ago
Yes thank you I was wondering exactly this. Quite reasonable size one might say.
eternityforest 3y ago
What are they using to compress tiles? Could they get even smaller with AVIF? Or AVIF+SVG overlaid for text?

pmontra 3y ago

It's 107 GB now. It would be great to be able to download only diffs and apply them to a daily, weekly, monthly baseline.

blutack 3y ago
Hmm I was just going off the Firefox estimate, maybe there's some browser difference?
- bdon 3y ago
  The newest major version has moved to a daily build channel (the last major version was last updated in April) Today 10/23 is 107GB. https://maps.protomaps.com/builds/ https://maps.protomaps.com/builds/ I'll update the documentation page today.
  - bdon 3y ago
    (I've updated the page at https://docs.protomaps.com/basemaps/downloads https://docs.protomaps.com/basemaps/downloads)
mycall 3y ago
I wonder if the compressed Hilbert ordering inside the tiles make them hard to diff efficiently.
- bdon 3y ago
  The ordering applies across tiles, and not within - it should be efficient to diff two archives because there is a single, well-defined ordering over the entire archive, and writing algorithms over the tileset is all 1D operations instead of 2D (a single zoom level plane) or 3D (multiple planes)

dinkblam 3y ago

thanks for the info, weird that this isn't answered in the FAQ

edhelas 3y ago

Exactly the kind of thing that could be downloaded using Bittorrent!

ElectricalUnion 3y ago
I assume that would require BitTorrent v2 (and I think more specifically something along BEP-46 - Updating Torrents Via DHT Mutable Items) or your map is gonna be out-of-sync and sad after a while.
- nextaccountic 3y ago
  are the torrents in the wild that uses this? and are there bittorrent clients that implement this? how do they surface the mutable aspect to the user? (is it like, your torrent client will silently update files, and if you dont want to update anymore you need to pause? can you still seed if you dont want to update?)
  - anacrolix 3y ago
    I use it in anacrolix/btlink. There's support in anacrolix/dht.
    - nextaccountic 3y ago
      That's very cool! Does this serve decentralized "web app" through bittorrent? Do I need your client to seed it, or can something like qbittorrent seed it? Will I seed only the last version of the page or all versions? Can someone download an earlier version or list all past versions?
  - acalcutt 3y ago
    I've been testing a setup that automatically generates OpenStreetMap/OpenMapTiles formatted pmtiles and mbtiles, then makes them available by torrent. The torrents get placed in a rss feed at https://planetgen.wifidb.net/ https://planetgen.wifidb.net/ Basically what I did was set up 'qbittorent' client to watch the openstreetmap rss weekly torrent feed. when it downloads a new pbf file, it runs this script I made at https://github.com/acalcutt/PlanetilerTorrent https://github.com/acalcutt/PlanetilerTorrent , which was based of the osm torrent creation process. The script creates pmtiles and mbtiles using planetiler, then makes torrent and starts seeding them in qbittorent. In qbittorent I have options like how long I want to seed for (30 days), what do do when done seeding (Delete the files) and also speed controls so I can limit bandwidth during my working hours. I have this running on a old laptop with 2TB nvme/64GB memory. It seeems to work pretty well so far. It would be nice if my internet speeds were a little better for initial seeding, but at least the torrents share the load with other people who are downloading/

bambax 3y ago

Yes thank you I was wondering exactly this. Quite reasonable size one might say.

eternityforest 3y ago

What are they using to compress tiles? Could they get even smaller with AVIF? Or AVIF+SVG overlaid for text?