2 ms·

Thanks for sharing, very intersting! I'm thinking the same wrt dropping Parquet. I don't need concurrent writes, which seems to me about the only true caveat

Thanks for sharing, very intersting! I'm thinking the same wrt dropping Parquet. I don't need concurrent writes, which seems to me about the only true caveat DuckDB would have. Two other questions I am asking myself: 1) Is there a an upper file size limit in duckdb's native format where performance might degrade? 2) Are there significant performance degradations/ hard limits if I want to consolidate from X DuckDB's into a single one by programmatically attaching them all and pulling data in via a large `UNION ALL` query? Or would you use something like Polars to query over N DuckDB's?

kianN 2y ago
I can offer my experience with respect to your two questions, though my use case is likely atypical. 1) I haven't personally run into upper size limits to the point of non linear performance degradation. However, some caveats to that are (a) most my files are in the range of 2-10gb with a few topping out near 100gb. (b) I am running a single r6gd metal as the primary interface with this which has 512 gb of ram. So, essentially, any one of my files can fit into ram. Even given that setup, I will mention that I find myself hand tuning queries a lot more than I was with Spark. Since duckdb is meant to be more lightweight the query optimization engine is less robust. 2) I can't speak too much towards this use case. I haven't had any occasion to query across duckdb files. However, I experimented on top of delta lake between duckdb and polars and never really found a true performance case for polars in my (again atypical use case) set of test. But definitely worth doing your own benchmarking on your specific use case :)
- thenaturalist 2y ago
  Thanks for sharing! :)