4 ms·

I really like the idea. Why OMT instead of protomaps? The latter is clearly where the community is moving towards (albeit very slowly). I'm somewhat sceptical

I really like the idea. Why OMT instead of protomaps? The latter is clearly where the community is moving towards (albeit very slowly). I'm somewhat sceptical about the "free with no API keys" idea. I guess your service is not guaranteed to be up so no one too big will rely on it. But what if you start getting abuse or someone using them on some humongous site (e.g. one of those cheap restaurant email builders that always embed a map), and you start getting way too much traffic from random sources and websites. What would you do?

l3x 2y ago
From the FAQs on GitHub [1] > What about PMTiles? > I would have loved to use PMTiles; they are a brilliant idea! > Unfortunately, making range requests in 80 GB files just doesn't work in production. It is fine for files smaller than 500 MB, but it has terrible latency and caching issues for full planet datasets. > If PMTiles implements splitting to <10 MB files, it can be a valid alternative to running servers. [1] https://github.com/hyperknot/openfreemap https://github.com/hyperknot/openfreemap
- apitman 2y ago
  That's an interesting claim. I make range requests to 100GB+ files (genomics) all the time for work and it works great. I've never considered total file size as directly related to latency in this respect, assuming you have some sort of an index of course.
  - bdon 2y ago
    You can test this claim directly against a AWS S3 bucket. First 100KB of a 100GB+ file: curl -H "Range: bytes=0-100000" https://overturemaps-tiles-us-west-2-beta.s3.amazonaws.com/2024-07-22/buildings.pmtiles https://overturemaps-tiles-us-west-2-beta.s3.amazonaws.com/2... --output tmp -w "%{time_total}" First 100KB at the 100GB mark: curl -H "Range: bytes=100000000000-100000100000" https://overturemaps-tiles-us-west-2-beta.s3.amazonaws.com/2024-07-22/buildings.pmtiles https://overturemaps-tiles-us-west-2-beta.s3.amazonaws.com/2... --output tmp -w "%{time_total}"
    - deleted 2y ago
      [deleted]
  - hyperknot 2y ago
    Here the requests are really really small, on average 405 bytes each. I guess in your genomics work you are making larger requests, so probably it's not so much of an issue. BTW, we are discussing latency with bdon in this issue, it seems to be specific to Cloudflare: https://github.com/hyperknot/openfreemap/issues/16 https://github.com/hyperknot/openfreemap/issues/16
    - apitman 2y ago
      I just tried @bmon's curl examples above with 100 byte requests. Similar results. I think the Cloudflare explanation is more likely.
- tobilg 2y ago
  If you store the PMTiles in S3 or any other object store that supports HTTP Range Requests, that's a no-brainer... In a normal disk on you own server, this might become interesting, yes.
- deleted 2y ago
  [deleted]
- mistrial9 2y ago
  ok except "full planet datasets" make little sense for terrestrial features. Splitting .. aka sharding the files into basic continents would make SO much sense. Asia is big, but no requests for Africa mixed in.. Australia would be manageable?
  - hyperknot 2y ago
    PMTiles could come up with a version in the future where instead of one 90 GB file, they have 9 thousand 10 MB files. That would work well I believe.
    - bdon 2y ago
      The latency for small files and ranges of large files is pretty similar on most storage platforms, but there are some exceptions like Cloudflare R2. The main reason PMTiles is one file and not two or more files is that it enables atomic updates in-place (which every mature storage platform supports) as well as ETag content change detection in downstream caches. All of the server and serverless implementations at http://github.com/protomaps http://github.com/protomaps support this now for AWS, S3-compatible storage, Google Cloud, and Azure.
      - wcedmisten 2y ago
        Now I'm curious, what causes the latency for range requests with R2?
        bdon 2y ago
        I don't have any insight into this other than observing how their storage system works, but here's some scripts I made last year to test: https://github.com/bdon/cloudflare-r2-latency https://github.com/bdon/cloudflare-r2-latency
        mannyv 2y ago
        Range requests means work and logic. Getting a file requires no logic. Also, I'm pretty sure range requests are going to be difficult to cache. That implies going to origin every request which is bad.
- bdon 2y ago
  See my response here: https://news.ycombinator.com/item?id=41638031 https://news.ycombinator.com/item?id=41638031
hyperknot 2y ago
So Protomaps is a really full stack map platform, they are competing with kind of like 3-4 projects at once: - There is the Protomaps schema. It's competing with OpenMapTiles and Shortbread (https://shortbread-tiles.org/ https://shortbread-tiles.org/) - There is the Protomaps file format. It's competing with MBTiles and say Btrfs images in OpenFreeMap. - There is the Protomaps styles. It's competing with OSM Liberty and OSM Bright styles (https://github.com/maputnik/osm-liberty https://github.com/maputnik/osm-liberty) - There is the way to host the tiles, on CloudFlare workers, range requests, etc. It's competing with MBTiles servers or OpenFreeMap's nginx servers. About getting abuse, how do you imagine it gets abuse? It's a static file showing OpenStreetMap data. I mean traffic is the only thing what can be a problem, but I think Hetzner should be good till 30-100 TB per server.
- bauruine 2y ago
  I did 150TB+ on a 40 Euro Hetzner server for years without any issue and also did the same amount in a similar price range on multiple different providers. The cloud and CDNs charge a huge markup.
  - hyperknot 2y ago
    That's great to know! I know of one story where someone got terminated on Hetzner because of the bandwidth but it turned out he was also running a Tor node.
- bdon 2y ago
  1) The Protomaps schema is mostly a re-implementation of the Tilezen project https://tilezen.readthedocs.io/en/latest/ https://tilezen.readthedocs.io/en/latest/ which is a linux foundation project. OpenMapTiles, which OpenFreeMap uses, while open source, does not have a license that encourages derivative works or enables distributing styles under a standard FOSS software license (https://www.npmjs.com/package/protomaps-themes-base https://www.npmjs.com/package/protomaps-themes-base). That's also one motivation for developing Shortbread which is at this point less developed than Tilezen. 2) The file format (PMTiles) addresses a different audience than either MBTiles or Btrfs images. Both of those require administering a server for tiles, while PMTiles requires static blob storage and nothing else. You do have the option of using a server like MBTiles/btrfs which ought to be comparable in latency, and that's documented here: https://docs.protomaps.com/deploy/ https://docs.protomaps.com/deploy/ as well as Lambda, Cloudflare Workers, Google Cloud Run and Azure Serverless functions. 3) There are no existing styles for MapLibre GL that work off the Tilezen layer, generalization and tagging scheme, so we need to develop one style, with multiple themes.

l3x 2y ago

From the FAQs on GitHub [1] > What about PMTiles? > I would have loved to use PMTiles; they are a brilliant idea! > Unfortunately, making range requests in 80 GB files just doesn't work in production. It is fine for files smaller than 500 MB, but it has terrible latency and caching issues for full planet datasets. > If PMTiles implements splitting to <10 MB files, it can be a valid alternative to running servers. [1] https://github.com/hyperknot/openfreemap https://github.com/hyperknot/openfreemap

apitman 2y ago
That's an interesting claim. I make range requests to 100GB+ files (genomics) all the time for work and it works great. I've never considered total file size as directly related to latency in this respect, assuming you have some sort of an index of course.
- bdon 2y ago
  You can test this claim directly against a AWS S3 bucket. First 100KB of a 100GB+ file: curl -H "Range: bytes=0-100000" https://overturemaps-tiles-us-west-2-beta.s3.amazonaws.com/2024-07-22/buildings.pmtiles https://overturemaps-tiles-us-west-2-beta.s3.amazonaws.com/2... --output tmp -w "%{time_total}" First 100KB at the 100GB mark: curl -H "Range: bytes=100000000000-100000100000" https://overturemaps-tiles-us-west-2-beta.s3.amazonaws.com/2024-07-22/buildings.pmtiles https://overturemaps-tiles-us-west-2-beta.s3.amazonaws.com/2... --output tmp -w "%{time_total}"
  - deleted 2y ago
    [deleted]
- hyperknot 2y ago
  Here the requests are really really small, on average 405 bytes each. I guess in your genomics work you are making larger requests, so probably it's not so much of an issue. BTW, we are discussing latency with bdon in this issue, it seems to be specific to Cloudflare: https://github.com/hyperknot/openfreemap/issues/16 https://github.com/hyperknot/openfreemap/issues/16
  - apitman 2y ago
    I just tried @bmon's curl examples above with 100 byte requests. Similar results. I think the Cloudflare explanation is more likely.
tobilg 2y ago
If you store the PMTiles in S3 or any other object store that supports HTTP Range Requests, that's a no-brainer... In a normal disk on you own server, this might become interesting, yes.
deleted 2y ago
[deleted]
mistrial9 2y ago
ok except "full planet datasets" make little sense for terrestrial features. Splitting .. aka sharding the files into basic continents would make SO much sense. Asia is big, but no requests for Africa mixed in.. Australia would be manageable?
- hyperknot 2y ago
  PMTiles could come up with a version in the future where instead of one 90 GB file, they have 9 thousand 10 MB files. That would work well I believe.
  - bdon 2y ago
    The latency for small files and ranges of large files is pretty similar on most storage platforms, but there are some exceptions like Cloudflare R2. The main reason PMTiles is one file and not two or more files is that it enables atomic updates in-place (which every mature storage platform supports) as well as ETag content change detection in downstream caches. All of the server and serverless implementations at http://github.com/protomaps http://github.com/protomaps support this now for AWS, S3-compatible storage, Google Cloud, and Azure.
    - wcedmisten 2y ago
      Now I'm curious, what causes the latency for range requests with R2?
      - bdon 2y ago
        I don't have any insight into this other than observing how their storage system works, but here's some scripts I made last year to test: https://github.com/bdon/cloudflare-r2-latency https://github.com/bdon/cloudflare-r2-latency
      - mannyv 2y ago
        Range requests means work and logic. Getting a file requires no logic. Also, I'm pretty sure range requests are going to be difficult to cache. That implies going to origin every request which is bad.
bdon 2y ago
See my response here: https://news.ycombinator.com/item?id=41638031 https://news.ycombinator.com/item?id=41638031

hyperknot 2y ago

So Protomaps is a really full stack map platform, they are competing with kind of like 3-4 projects at once: - There is the Protomaps schema. It's competing with OpenMapTiles and Shortbread (https://shortbread-tiles.org/ https://shortbread-tiles.org/) - There is the Protomaps file format. It's competing with MBTiles and say Btrfs images in OpenFreeMap. - There is the Protomaps styles. It's competing with OSM Liberty and OSM Bright styles (https://github.com/maputnik/osm-liberty https://github.com/maputnik/osm-liberty) - There is the way to host the tiles, on CloudFlare workers, range requests, etc. It's competing with MBTiles servers or OpenFreeMap's nginx servers. About getting abuse, how do you imagine it gets abuse? It's a static file showing OpenStreetMap data. I mean traffic is the only thing what can be a problem, but I think Hetzner should be good till 30-100 TB per server.

bauruine 2y ago
I did 150TB+ on a 40 Euro Hetzner server for years without any issue and also did the same amount in a similar price range on multiple different providers. The cloud and CDNs charge a huge markup.
- hyperknot 2y ago
  That's great to know! I know of one story where someone got terminated on Hetzner because of the bandwidth but it turned out he was also running a Tor node.
bdon 2y ago
1) The Protomaps schema is mostly a re-implementation of the Tilezen project https://tilezen.readthedocs.io/en/latest/ https://tilezen.readthedocs.io/en/latest/ which is a linux foundation project. OpenMapTiles, which OpenFreeMap uses, while open source, does not have a license that encourages derivative works or enables distributing styles under a standard FOSS software license (https://www.npmjs.com/package/protomaps-themes-base https://www.npmjs.com/package/protomaps-themes-base). That's also one motivation for developing Shortbread which is at this point less developed than Tilezen. 2) The file format (PMTiles) addresses a different audience than either MBTiles or Btrfs images. Both of those require administering a server for tiles, while PMTiles requires static blob storage and nothing else. You do have the option of using a server like MBTiles/btrfs which ought to be comparable in latency, and that's documented here: https://docs.protomaps.com/deploy/ https://docs.protomaps.com/deploy/ as well as Lambda, Cloudflare Workers, Google Cloud Run and Azure Serverless functions. 3) There are no existing styles for MapLibre GL that work off the Tilezen layer, generalization and tagging scheme, so we need to develop one style, with multiple themes.