4 ms·
Apache Iceberg
- deleted 3y ago[deleted]
- appplication 3y agoIceberg is really quite cool. Having an open big data storage format that also approaches qualities similar to RDBMS is a very nice improvement from what came before (e.g. raw parquet/avro etc). I’m excited to see how this continues to evolve. It’s a shame Databricks decided to compete with delta lake rather than joining to add features to iceberg.
- waffletower 3y agoAWS has been steadily integrating Iceberg support into Athena (based on Trino) and Redshift Spectrum. The practical advantages of schema evolution, time travel, and update and delete queries, form a convincing alternative to use of AWS Glue crawlers in my experience. There are open source Kafka Connect connectors available as well: https://github.com/tabular-io/iceberg-kafka-connect https://github.com/tabular-io/iceberg-kafka-connect https://github.com/getindata/kafka-connect-iceberg-sink https://github.com/getindata/kafka-connect-iceberg-sink
- nextworddev 3y agoIceberg is pushed to hedge against Delta / Databricks
- Reubend 3y agoShould I think of Iceberg as the on-disk equivalent of Arrow?
- adammarples 3y agoParquet is basically on disk arrow. Iceberg is a metadata layer above that that lets you group parquet files into tables, and treat them like a database by keeping track of what changes you make and writing them into new files.