5 ms·
Do you still use file formats at all in your work? I'm currently thinking of ditching Parquet all together, and going all in DuckDB files. I don't need concur
by thenaturalist 2y ago
Do you still use file formats at all in your work?
I'm currently thinking of ditching Parquet all together, and going all in DuckDB files.
I don't need concurrent writes, my data would rarely exceed 1TB and if it were, I could still offload to Parquet.
Conceptually I can't see a reason for this not working, but given the novelty of the tech I'm wondering if it'll hold up.
- RobinL 2y agoWe're still using parquet. So we use the native duckdb format for intermediate processing (during pipeline execution) but the end results are saved out as parquet. This is partly because customers often read the data from other tools (e.g. AWS athena) I'd be interested in hearing about experiences of using duckdb files though, i can see instances where it could be useful to us