4 ms·

Funny enough, we started out with a file-based system like the one you described, and moved to an HDF5-based system. Are you not concerned about i-node consump

• nimrody 11y ago
  It depends on whether you have many small files or a small number of large files. In the former case, one file approach (HDF5) makes sense. In the latter case, you don't have a problem with i-node consumption and you gain the ability to easily access only the data you need without bringing the entire data-set into memory. I kind of the like the approach suggested above - if you have many small files store them in a zip archive and use some library to access the data directly.