5 ms·

The author (my colleague, and probably the most talented developer I know) isn't replacing HDF5 with a 'proprietary binary format': in fact, the transition is a

by nippoo 11y ago

The author (my colleague, and probably the most talented developer I know) isn't replacing HDF5 with a 'proprietary binary format': in fact, the transition is as simple as replacing "HDF5 group" with "folder in a filesystem", "HDF5 dataset" with "binary file on the filesystem" (ie you store each array item sequentially on disk, exactly as HDF5 or any other format will store it, which you can memmap trivially with any programming language), and "HDF5 attribute" with "JSON / XML / INI / text file". "When in the field and the system shits the bed", to quote you... you can just open the dataset in Windows Explorer. Or Mac OS Finder. Or Nautilus. Or using 'cd' and 'ls' in Linux. Want to look at an array? Sure, open it in a hex editor or Python or, heck, FORTRAN88. You can .tar up your folder (or subfolder, or any arbitrary subset of the dataset) and send it to someone with no knowledge of the format whatsoever, and they'll be able to make sense of it in minutes. This isn't anything remotely complex - it's just using the filesystem rather than creating a filesystem-within-a-filesystem. Want to keep track of changes in your massive dataset? Sure, just back it up on Mac OS X Time Machine, or a simple rsync script, or even Git LFS. Most researchers aren't computer scientists; they know how to use Dropbox and Notepad++ and open text files, and they don't want to have to install a Java-based HDF5View when they could just use Windows Explorer. It's not even that HDF5 is that bad, it's just that filesystems are, in many respects, so much better. (If it wasn't clear from the article, we're not just misinformed - we're making this call after having developed an entire software suite around HDF5, spent about two years of firefighting HDF5 issues and wasted days of development time (so many horror stories) - this is actual feedback from several dozen users, thousands of datasets, and petabytes of data.)

ipunchghosts 11y ago
Filesystems are the worst! Try telling a customer to tar up a directory and send it to you -- things get lost so easily! Most of our customers dont even know what "tar" means. I think you are asking for trouble going with a filesystem as a storage mechanism. HDFfView is not the only viewer in town. There are several viewers. I've used HDF5 weekly for the last 5 years and so have my associates and its been wonderful. MATLAB even uses it to store its .mat files these days. I think you are misinformed and Im sticking to my story !
- nippoo 11y ago
  It depends who your users are and what needs they have. Our users are mostly MATLAB/Python users, pretty tech-savvy, and being able to edit individual bits of the dataset with other applications or write their own code to analyse them is an often-used feature. It's rare we ever need the whole dataset - in fact, it's really great to be able to say to the user "don't send us your 100GB dataset: just go to the "acquisition" subfolder and send me the 10MB file called "oscilloscope.dat"". With HDF5 this is difficult enough that it's almost always easier to send 99.9% of useless data (i.e. the whole file) when all you want is a single array within it. If your users will rarely need to do this, you could just store the entire folder hierarchy in a .zip and access it using standard tools that most programming languages have. It's worth noting that the new Microsoft Office formats do exactly this - in their case, a bunch of XML files inside a .ZIP. (Rename a .docx to .zip and you'll see!). MATLAB has moved from their own custom binary format to HDF5, which is the lesser of two evils.
  - ipunchghosts 11y ago
    Your usecase doesnt seem like HDF5 would be a good fit. Yes, I do know about docx and zip. I'm happy with what matlab has done. People send me .mat files and I happily process them in python. And my plots usually look much nicer also. :)
deleted 11y ago
[deleted]
echlebek 11y ago
Funny enough, we started out with a file-based system like the one you described, and moved to an HDF5-based system. Are you not concerned about i-node consumption with the file-system based approach?
- nimrody 11y ago
  It depends on whether you have many small files or a small number of large files. In the former case, one file approach (HDF5) makes sense. In the latter case, you don't have a problem with i-node consumption and you gain the ability to easily access only the data you need without bringing the entire data-set into memory. I kind of the like the approach suggested above - if you have many small files store them in a zip archive and use some library to access the data directly.

by nippoo 11y ago

ipunchghosts 11y ago

Filesystems are the worst! Try telling a customer to tar up a directory and send it to you -- things get lost so easily! Most of our customers dont even know what "tar" means. I think you are asking for trouble going with a filesystem as a storage mechanism. HDFfView is not the only viewer in town. There are several viewers. I've used HDF5 weekly for the last 5 years and so have my associates and its been wonderful. MATLAB even uses it to store its .mat files these days. I think you are misinformed and Im sticking to my story !

nippoo 11y ago
It depends who your users are and what needs they have. Our users are mostly MATLAB/Python users, pretty tech-savvy, and being able to edit individual bits of the dataset with other applications or write their own code to analyse them is an often-used feature. It's rare we ever need the whole dataset - in fact, it's really great to be able to say to the user "don't send us your 100GB dataset: just go to the "acquisition" subfolder and send me the 10MB file called "oscilloscope.dat"". With HDF5 this is difficult enough that it's almost always easier to send 99.9% of useless data (i.e. the whole file) when all you want is a single array within it. If your users will rarely need to do this, you could just store the entire folder hierarchy in a .zip and access it using standard tools that most programming languages have. It's worth noting that the new Microsoft Office formats do exactly this - in their case, a bunch of XML files inside a .ZIP. (Rename a .docx to .zip and you'll see!). MATLAB has moved from their own custom binary format to HDF5, which is the lesser of two evils.
- ipunchghosts 11y ago
  Your usecase doesnt seem like HDF5 would be a good fit. Yes, I do know about docx and zip. I'm happy with what matlab has done. People send me .mat files and I happily process them in python. And my plots usually look much nicer also. :)

deleted 11y ago

[deleted]

echlebek 11y ago

Funny enough, we started out with a file-based system like the one you described, and moved to an HDF5-based system. Are you not concerned about i-node consumption with the file-system based approach?

nimrody 11y ago
It depends on whether you have many small files or a small number of large files. In the former case, one file approach (HDF5) makes sense. In the latter case, you don't have a problem with i-node consumption and you gain the ability to easily access only the data you need without bringing the entire data-set into memory. I kind of the like the approach suggested above - if you have many small files store them in a zip archive and use some library to access the data directly.