10 ms·
If you convert those PBFs to Parquet files. You can then use Duckdb to search them with sub-sec response. Plus you get the added bonus of being able to host in
by DonnyV 2y ago
If you convert those PBFs to Parquet files. You can then use Duckdb to search them with sub-sec response. Plus you get the added bonus of being able to host in an S3 type cloud storage.
https://duckdb.org/docs/extensions/httpfs/s3api https://duckdb.org/docs/extensions/httpfs/s3api
- highway900 2y agoThis is the business. I highly recommend this solution. Streaming queries via duckdb or polars will use little to memory and be lightning fast for very large datasets.
- jtarchie 2y agoThat's an interesting take. I also found [flatgeobuf](https://github.com/flatgeobuf/flatgeobuf https://github.com/flatgeobuf/flatgeobuf), which promises something similar-ish. Perhaps not the same interface for querying, but organizing data in an r-tree format for streaming from a blob store.