3 ms·

Filesystems are the worst! Try telling a customer to tar up a directory and send it to you -- things get lost so easily! Most of our customers dont even know w

Filesystems are the worst! Try telling a customer to tar up a directory and send it to you -- things get lost so easily! Most of our customers dont even know what "tar" means. I think you are asking for trouble going with a filesystem as a storage mechanism. HDFfView is not the only viewer in town. There are several viewers. I've used HDF5 weekly for the last 5 years and so have my associates and its been wonderful. MATLAB even uses it to store its .mat files these days. I think you are misinformed and Im sticking to my story !

nippoo 11y ago
It depends who your users are and what needs they have. Our users are mostly MATLAB/Python users, pretty tech-savvy, and being able to edit individual bits of the dataset with other applications or write their own code to analyse them is an often-used feature. It's rare we ever need the whole dataset - in fact, it's really great to be able to say to the user "don't send us your 100GB dataset: just go to the "acquisition" subfolder and send me the 10MB file called "oscilloscope.dat"". With HDF5 this is difficult enough that it's almost always easier to send 99.9% of useless data (i.e. the whole file) when all you want is a single array within it. If your users will rarely need to do this, you could just store the entire folder hierarchy in a .zip and access it using standard tools that most programming languages have. It's worth noting that the new Microsoft Office formats do exactly this - in their case, a bunch of XML files inside a .ZIP. (Rename a .docx to .zip and you'll see!). MATLAB has moved from their own custom binary format to HDF5, which is the lesser of two evils.
- ipunchghosts 11y ago
  Your usecase doesnt seem like HDF5 would be a good fit. Yes, I do know about docx and zip. I'm happy with what matlab has done. People send me .mat files and I happily process them in python. And my plots usually look much nicer also. :)

nippoo 11y ago

It depends who your users are and what needs they have. Our users are mostly MATLAB/Python users, pretty tech-savvy, and being able to edit individual bits of the dataset with other applications or write their own code to analyse them is an often-used feature. It's rare we ever need the whole dataset - in fact, it's really great to be able to say to the user "don't send us your 100GB dataset: just go to the "acquisition" subfolder and send me the 10MB file called "oscilloscope.dat"". With HDF5 this is difficult enough that it's almost always easier to send 99.9% of useless data (i.e. the whole file) when all you want is a single array within it. If your users will rarely need to do this, you could just store the entire folder hierarchy in a .zip and access it using standard tools that most programming languages have. It's worth noting that the new Microsoft Office formats do exactly this - in their case, a bunch of XML files inside a .ZIP. (Rename a .docx to .zip and you'll see!). MATLAB has moved from their own custom binary format to HDF5, which is the lesser of two evils.

ipunchghosts 11y ago
Your usecase doesnt seem like HDF5 would be a good fit. Yes, I do know about docx and zip. I'm happy with what matlab has done. People send me .mat files and I happily process them in python. And my plots usually look much nicer also. :)