4 ms·
Apache Iceberg now supports geospatial data types natively
- ddkto 2y agoSome more details: https://cloudnativegeo.org/blog/2025/02/geoparquet-2.0-going-native/ https://cloudnativegeo.org/blog/2025/02/geoparquet-2.0-going...
- cratermoon 2y agoThis is almost what I want, as it mentions the data types GEOMETRY and GEOGRAPHY, but is there a pointer somewhere to the definitions of the types?
- gloflo 2y agoClick through the links on that page.
- cratermoon 2y agoI did. The links are giving me information about as useful as "a monad is a monoid in the category of endofunctors, what's the problem?". In fact, your comment is about equally helpful. Ever heard the phrase "technically accurate but totally useless"? https://wiki.c2.com/?UselessTruth https://wiki.c2.com/?UselessTruth
- dr-jia-yu 2y agoWherobots wrote a detailed tech review of the new types. You can find the definition and reasoning there: https://wherobots.com/iceberg-geo-technical-insights-and-implementation-strategies/ https://wherobots.com/iceberg-geo-technical-insights-and-imp...
- cratermoon 2y agoBetter. I still feel like there's an element of explanations there nearly as opaque as "a monad is a monoid in the category of endofunctors", but I can mostly forgive it. Clearly the review is written for someone already familiar with GIS terms. For example, "The geometry type represents spatial objects in a planar space using Cartesian geometry, assuming all calculations, including distance and area measurements, are performed on a flat surface." says nothing that couldn't be better stated as "this is for flat-projection maps, like the Mercator".
- bitschubser_ 2y agoI’m super exited about the planned support for multidimensional data, I know zarr is there but a long term storage format other than netcdf would be interesting maybe also something to replace grib, geotiff etc to share files
- Upitor 2y agoWhere do you see the upcoming support for multidim data? Link?
- bitschubser_ 2y ago> This is just the beginning of modernizing geospatial data storage. We’re already looking ahead to other types of geospatial data such as raster, point cloud, spatial indexes… its not far from raster to full multidimensional
- tomnicholas1 2y agoSurely Zarr is already a long-term storage format for multidimensional data? It can even be mapped directly to netCDF, GRIB and geoTIFF via VirtualiZarr[0]. Also if you like Iceberg and you like arrays you will really like Icechunk[1], which is Version-controlled Zarr! [0] https://github.com/zarr-developers/VirtualiZarr https://github.com/zarr-developers/VirtualiZarr [1] https://icechunk.io/en/latest/ https://icechunk.io/en/latest/
- bitschubser_ 2y agoI know icechunk and I’m a huge fan of earthmover. But a common binary format like parquet seems nice… with interop for e.g duckdb and geo queries, you can “just load” era5 and do something like get wind direction/speed along the following path for the last 5 years group by day etc…
- lysecret 2y agoIf you know the exact tensor shape of your data ahead of time Zarr works well (we use it as the dataformat for our ml experiments). If you have dynamically growing data or irregular shapes zarr doesn't work as well.
- ZeroCool2u 2y agoThis is such a big deal. Trying to get stuff done with geospatial data in native formats in big companies has been somewhat painful for a while, because a lot of them tend to default to using DataBricks and Delta format. At least now there's a great open source tech stack combo with Trino as your query engine (which is decidedly a lot less annoying to run at scale than OSS Spark) and Iceberg as your storage format that a lot of your "Enterprise Architecture" types can be comfortable enough with.
- tiems 2y agoSince it’s getting added to the parquet spec itself, it should hopefully make its way to delta too soon.
- Mosarwat 2y agoPlease note that not all query engines supports the native geo type in Iceberg yet. The first one to support it is Apache Sedona, which works well with Spark: https://github.com/apache/sedona https://github.com/apache/sedona However, the ultimate goal is to make more engines (e.g., Arrow, Trino...) support the geo type too
- bz_bz_bz 2y agoAnyone know if this means Arrow could be next? I assume GeoParquet is no longer needed; I wonder what happens to GeoArrow.
- Mosarwat 2y agoGeoParquet will still be used for bit since both the native geo types in parquet and geoparquet have slight differences. But, the ultimate goal is to shift all data / workloads to the parquet native geo in a couple of years!
- Mosarwat 2y agoNow, Apache Sedona is the first engine that will support that native geo type in parquet, but Arrow will also support it very soon.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- dr-jia-yu 2y agoA more detailed tech review (from Wherobots) of 2 new Geo types can be found here: Iceberg GEO: Technical Insights and Implementation Strategies https://wherobots.com/iceberg-geo-technical-insights-and-implementation-strategies/ https://wherobots.com/iceberg-geo-technical-insights-and-imp...