3 ms·
Most of my data is too big for CSVs but too small to justify distributed storage. I use HDF5 with chunking and column compression. I think many other people in
by iskander 14y ago
Most of my data is too big for CSVs but too small to justify distributed storage. I use HDF5 with chunking and column compression. I think many other people in the sciences and finance also do this (along with using NetCDF).