3 ms·
I've found that for tabular data parquet can't be beat, the only time textual formats are superior are for human readabilty (which imo can be solved by tooling)
by theLiminator 3y ago
I've found that for tabular data parquet can't be beat, the only time textual formats are superior are for human readabilty (which imo can be solved by tooling)
- hermitcrab 3y agoIt is useful to be able to create, view or edit data in a standard text editor though.
- theLiminator 3y agoEdits might be a little tricky, there's no concept of writing in place in parquet afaict. But there's absolutely no reason why reads/creation can't be supported in text editors (I'm sure some do).
- hermitcrab 3y agoI often have to create test data files a few rows long. Very easy to do for a CSV file.
- wenc 3y agoAbsolutely. This is also my use case where I have GBs of data in S3 as Parquet and a small CSV file that I have to join them to. And DuckDB reads CSVs and Parquet and SQLite and others. I can join all these heterogeneous data types in a single SQL statement and have the assurance that it’ll be done correctly. I believe clickhouse-local can do the same.