4 ms·
The main use case for me for DuckDB is in Data Lake-related tasks where cloud providers often do not provide great/cost-effective/otherwise limited services. I
by tobilg 3y ago
The main use case for me for DuckDB is in Data Lake-related tasks where cloud providers often do not provide great/cost-effective/otherwise limited services.
In the case of AWS, repartitioning Parquet files in S3 via Athena CTAS statements in limited to 100 active partitions, which is a bummer to work around. Therefore, I’m using DuckDB with repartitioning queries, because it doesn’t have the 100 partition limit.
I wrote a blog post about it at https://tobilg.com/casual-data-engineering-or-a-poor-mans-data-lake-in-the-cloud-part-i https://tobilg.com/casual-data-engineering-or-a-poor-mans-da... Additionally, to get started with using DuckDB serverlessly in Lambda functions, you can have a look at https://tobilg.com/using-duckdb-in-aws-lambda https://tobilg.com/using-duckdb-in-aws-lambda
- lysecret 3y agoWe extensively explored this approach for a use-case and dropped it in favour of BigQuery external data query for one core reason: It can't do streaming reads if you have to do any kind of operation on the underlying data, for almost everything it will have to load all the relevant parquet files locally to do the operations. You might have been able to solve this using row partitions on the underlying files beforehand, but it was too late for us. Also, it actually turned out to be much more expensive to run the operations in cloud run, than relying on BQ (due to the fact it wouldn't stream as we wished). More than anything it gave me renewed appreciation for BQ.
- chrisjc 3y agoVery interesting. Would you mind mentioning which external table format you were using? Iceberg by any chance? I wonder if there wasn't a middle-ground that could still utilize DuckDB or similar. You still rely on BQ to deliver the subset of data needed, and then rely on DuckDB to do the "last-mile" analytics: grouping, filtering, ordering, etc...?
- tobilg 3y agoI don't think this is correct, or I didn't really understand your use case. Could you give an example for "any kind of operation on the underlying data"?