4 ms·
I've been following DuckDB for a while now, and even tinkered with a layer on top called "IceDB" (totally needs a rewrite: https://blog.danthegoodman.com/introd
by dangoodmanUT 3y ago
I've been following DuckDB for a while now, and even tinkered with a layer on top called "IceDB" (totally needs a rewrite: https://blog.danthegoodman.com/introducing-icedb--a-serverless-json-analytics-database https://blog.danthegoodman.com/introducing-icedb--a-serverle...)
The issue I see now is that there is no good way to know what files will match well when reading from remote (decoupled) storage.
While it does support hive partitioning (thank god), and S3 list calls, if you are looking at doing inserts frequently you need some way to merge these parquet files.
The MergeTree engine is my favorite thing about ClickHouse, and why it's still my go-to. I think if there was a serverless way to merge parquet (which was the aim of IceDB) that would make DuckDB massively more powerful as a primary OLAP db.
- chrisjc 3y agoIceDB sounds like an interesting project. What are your thoughts on Apache Iceberg? Seems like a similar idea, and gaining a lot of traction in the data-eng world.
- LewisJEllis 3y agoYea, DuckDB is a slam dunk when you have a relatively static dataset - object storage is your durable primary SSOT, and ephemeral VMs running duckdb pointed at the object storage parquet files are your scalable stateless replicas - but the story gets trickier in the face of frequent ongoing writes / inserts. ClickHouse handles that scenario well, but I suspect the MotherDuck folks have answers for that in mind :)