4 ms·
Hi, in case you did not find the answer yet. In my hamble opinion: - choose Iceberg: If you have several computing/query engines other than Spark, like Presto,
by waterlx 2y ago
Hi, in case you did not find the answer yet. In my hamble opinion:
- choose Iceberg: If you have several computing/query engines other than Spark, like Presto, Flink. Iceberg has a great extraction and design for a engine-independent table format. But its learning cost is relative high
- choose Delta: If you only have Spark and would like to be deeply binded with Databricks
- choose Hudi: If you would like to use data lake out-of-the-box and it is quite easy to use.
- If your data is updated frequently, like streaming, check https://paimon.apache.org/ https://paimon.apache.org/ if you would like to be deeply binded with Flink
- indoordin0saur 2y agoThank you! Sounds like iceberg is the best then. I'm very allergic to lock-in. Currently we're very Spark heavy and our query engine is AWS Redshift Serverless. The recent AWS Glue Catalog support for Iceberg seems to make this promising.
- ruipds 2y agoI heard from a AWS worker that they consider Iceberg to be the future. A lot of their services will be glued together with it.