DuckDB use case? I remember being in the same spot. Then I realized "the" use case for DuckDB didn't really exist. There were dozens of use-cases. No wonder I c

DuckDB use case? I remember being in the same spot. Then I realized "the" use case for DuckDB didn't really exist. There were dozens of use-cases. No wonder I couldn't figure out "the" use case from reading all the articles/blogs/hype people were creating about what they were doing with DuckDB. It seemed like everyone was using it in different ways. It's a data engineering/analytics Swiss army knife. ETL/ELT, ad-hoc/exploratory, munger/wrangler, edge-analytics, and so on... > DuckDB will eventually hit a ceiling. Yeah, probably. Just about every database has hit a ceiling in the past. But then someone comes out with some fantastic new idea to overcome the challenges to some degree. Map-reduce, moving the query to the data instead of the data to the query, serverless, separating storage/compute... However, what if we start thinking along these lines with DuckDB? Reading parquet files addresses separating storage and compute. Parquet also provides columnar/row-grouped data giving us push-down predicates (so kinda moving part of the query closer to the data). We can run a DuckDb instance (EC2/S3) closer to the data so that sorta helps too. What I'm really excited about using DuckDB in a similar way to map-reduce. What if there was a way to take some SQL's logical plan and turn it into a physical plan that uses compute resources from a pool of (serverless) DuckDB instances. Starting at the leaves of the graph (physical plan) pulling/filtering data from the source (parquet files), and returning their completed work up the branches until it is completed and ready to be served up as results. I've seen a few examples of this already, but nothing that I would consider production ready. I have a hunch that someone is going to drop such a project on us shortly, and it's going to change a lot of things we have become use to in the data world. https://github.com/BauplanLabs/quack-reduce https://github.com/BauplanLabs/quack-reduce https://www.boilingdata.com/ https://www.boilingdata.com/