3 ms·

As a meta-topic, the standard formats for geospatial are virtually all deeply suboptimal to the point of being arguably broken for modern large-scale geospatial

As a meta-topic, the standard formats for geospatial are virtually all deeply suboptimal to the point of being arguably broken for modern large-scale geospatial analysis. What we have is a giant collection of narrow solutions to narrow problems, none of which efficiently interoperate with each other. It hasn’t modernized much at all, the tool chain is basically the same tech we had 25 years ago with a fresh coat of paint. Fixing this is a difficult and extremely deep engineering and computer science problem, which is why there has been so little progress. Most improvements have been driven by users, but they aren’t able to make the investments to fix the problems underneath their problem, they have other work to do. Any fundamental improvements would materially break existing workflows, so adoption would be slow at best. The entire spatial domain is stuck in a deep local minima.

sid_tf 2y ago
I share the same feelings, hence I along with OP, made this attempt of creating a library based on whatever seemed most performant and closest to regular data engineering world. 1. Geoparquet for metadata (iceberg catalog could be added later if scale is extremely large) 2. Cloud Optimized Geotiff (COG) for image data (which is what NASA and ESA have been pumping for last 5 years and still do) 3. and an efficient lightweight library that can quickly grab pieces of data from 100s or 1000s of raster files parallely. I did not want to attempt creating another format, coz I felt that we as geocommunity are at a stage where such tools and files put together forms a pretty good first level foundation to build on. No need for heavy legacy tools, closer to regular data (with parquet) and closer to duckdb world. I feel adoption may be good, if this is the thesis / guiding principles. Baby steps surely, but good steps I believe, and this is all thanks to the open source community.

sid_tf 2y ago

I share the same feelings, hence I along with OP, made this attempt of creating a library based on whatever seemed most performant and closest to regular data engineering world. 1. Geoparquet for metadata (iceberg catalog could be added later if scale is extremely large) 2. Cloud Optimized Geotiff (COG) for image data (which is what NASA and ESA have been pumping for last 5 years and still do) 3. and an efficient lightweight library that can quickly grab pieces of data from 100s or 1000s of raster files parallely. I did not want to attempt creating another format, coz I felt that we as geocommunity are at a stage where such tools and files put together forms a pretty good first level foundation to build on. No need for heavy legacy tools, closer to regular data (with parquet) and closer to duckdb world. I feel adoption may be good, if this is the thesis / guiding principles. Baby steps surely, but good steps I believe, and this is all thanks to the open source community.