3 ms·
I was almost going to build a lakehouse* with DuckDB because I low-key love it, easiest and strongest analytical engine I've found yet: scale from laptops to bi
by mrbungie 1y ago
I was almost going to build a lakehouse* with DuckDB because I low-key love it, easiest and strongest analytical engine I've found yet: scale from laptops to big metal, while being mostly out-of-core when doing sane stuff, and avoiding distributed computing for SQL in the process (looking at you Spark).
That is until I found out it does not support Iceberg writes[1], big nono as I would need another engine for inserts, and I want a simple stack :(. What a bummer.
[1] https://github.com/duckdb/duckdb_iceberg/issues/37 https://github.com/duckdb/duckdb_iceberg/issues/37
*that is what they are called now aren't they? I just can't follow the terms anymore haha.
- mritchie712 1y agoit's coming. they already have hive style parquet writes. Iceberg is more complicated than that, but it's certainly doable.
- mrbungie 1y agoYeah, it just would be great if it already did so and I hope it supports Iceberg soon, as it would enable me to change expensive (and bad) engines like AWS Athena for something more manageable. Don't get me wrong, I'm just being a tongue-in-check egotistical bastard data engineer from hell. DuckDB is a fine piece of software as it is, and those mantainers deserve heaven.
- jeadie 1y agoThis is one of the ideas behind using DuckDB in github.com/spiceai/spiceai
- mrbungie 1y agoLooks very cool! I will take a look, tysm!
- anentropic 1y agoThat looks like an amazing "swiss army knife"...!
- nicornk 1y agoFivetran tried to upstream write support but it was not accepted https://github.com/duckdb/duckdb-iceberg/pull/95 https://github.com/duckdb/duckdb-iceberg/pull/95
- shakna 1y agoThat sounds less "not accepted" and more "will implement, rewrite required". It was only a couple months ago.
- sukhavati 1y agosame here man, ended up going with trino explicitly for writing and data management and using chdb/duckdb to process data for front-ends etc (mostly ethereum data so chdb "support" for ui256 is quite important)
- buremba 1y agoNot just for building a new one, it can also complement existing data-warehouse/lakehouses: https://github.com/buremba/universql https://github.com/buremba/universql The flight extension is excellent as it removes the need to write C++ extensions and lets you use your favorite language to develop native DuckDB catalogs. It's straightforward to build data lake connectors and plug them in as a flight catalog, thanks to Airport!
- benrutter 1y agoI'm curious, did you consider delta tables? Pretty sure duckdb supports them nicely. If you did, how come you chose not to go with them?
- mrbungie 1y agoAfair (I might be wrong) AWS and a big chunk of the industry is promoting Iceberg over Delta. Delta is mostly backed up by Databricks.