4 ms·
For tabular data, parquet with zstd compression is very straight forward and quick. Columnar and split into row groups with stats on the columns within each, ea
by IanCal 4y ago
For tabular data, parquet with zstd compression is very straight forward and quick. Columnar and split into row groups with stats on the columns within each, easily processed by many tools as standard. You can go further into arrow but parquet alone solves the majority of key issues IMO for tabular data.
- kzrdude 4y agoParquet looks good but HDF5 is good just because it's hierarchical and can group together multiple related datasets. Maybe data taken in the same session on completely different axes/coordinates this is what I've missed with parquet.