4 ms·

Reading all these comments that bash HDF5 makes me want to tell how HDF5 has really worked for my group. Although the spec is huge, there is plenty of sample c

Reading all these comments that bash HDF5 makes me want to tell how HDF5 has really worked for my group. Although the spec is huge, there is plenty of sample code online to get it working. You do actually have to read it though to understand slabs, hyperslabs, strides etc. Once you get though, its really versitle. As far as speed, we used it to replace our propriatary data format. We would have to provide readers to all the scientists that use our data. It was nightmare. Some people want stuff in R, some in Python 2.7, some in Python 3.4, some in Matlab, and the list goes on. HDF5 gets rid of all this. When in the field and the system shits the bed, its really easy to open an HDF5 file in HDFview and inspect the file contents. I dont always have matlab available when im in the field, same with python. Sometimes I just need to look at a time series and I can diagnose the problems with the system. For me, its silly in 2016 to have any kind of proprietary binary format when something HDF5 exists. Many of the complaints the author had makes me think he the stereotypical scientist of really smart in one area but cant program worth the beans. I dont think that HDF5's fault.

nippoo 11y ago
The author (my colleague, and probably the most talented developer I know) isn't replacing HDF5 with a 'proprietary binary format': in fact, the transition is as simple as replacing "HDF5 group" with "folder in a filesystem", "HDF5 dataset" with "binary file on the filesystem" (ie you store each array item sequentially on disk, exactly as HDF5 or any other format will store it, which you can memmap trivially with any programming language), and "HDF5 attribute" with "JSON / XML / INI / text file". "When in the field and the system shits the bed", to quote you... you can just open the dataset in Windows Explorer. Or Mac OS Finder. Or Nautilus. Or using 'cd' and 'ls' in Linux. Want to look at an array? Sure, open it in a hex editor or Python or, heck, FORTRAN88. You can .tar up your folder (or subfolder, or any arbitrary subset of the dataset) and send it to someone with no knowledge of the format whatsoever, and they'll be able to make sense of it in minutes. This isn't anything remotely complex - it's just using the filesystem rather than creating a filesystem-within-a-filesystem. Want to keep track of changes in your massive dataset? Sure, just back it up on Mac OS X Time Machine, or a simple rsync script, or even Git LFS. Most researchers aren't computer scientists; they know how to use Dropbox and Notepad++ and open text files, and they don't want to have to install a Java-based HDF5View when they could just use Windows Explorer. It's not even that HDF5 is that bad, it's just that filesystems are, in many respects, so much better. (If it wasn't clear from the article, we're not just misinformed - we're making this call after having developed an entire software suite around HDF5, spent about two years of firefighting HDF5 issues and wasted days of development time (so many horror stories) - this is actual feedback from several dozen users, thousands of datasets, and petabytes of data.)
- ipunchghosts 11y ago
  Filesystems are the worst! Try telling a customer to tar up a directory and send it to you -- things get lost so easily! Most of our customers dont even know what "tar" means. I think you are asking for trouble going with a filesystem as a storage mechanism. HDFfView is not the only viewer in town. There are several viewers. I've used HDF5 weekly for the last 5 years and so have my associates and its been wonderful. MATLAB even uses it to store its .mat files these days. I think you are misinformed and Im sticking to my story !
  - nippoo 11y ago
    It depends who your users are and what needs they have. Our users are mostly MATLAB/Python users, pretty tech-savvy, and being able to edit individual bits of the dataset with other applications or write their own code to analyse them is an often-used feature. It's rare we ever need the whole dataset - in fact, it's really great to be able to say to the user "don't send us your 100GB dataset: just go to the "acquisition" subfolder and send me the 10MB file called "oscilloscope.dat"". With HDF5 this is difficult enough that it's almost always easier to send 99.9% of useless data (i.e. the whole file) when all you want is a single array within it. If your users will rarely need to do this, you could just store the entire folder hierarchy in a .zip and access it using standard tools that most programming languages have. It's worth noting that the new Microsoft Office formats do exactly this - in their case, a bunch of XML files inside a .ZIP. (Rename a .docx to .zip and you'll see!). MATLAB has moved from their own custom binary format to HDF5, which is the lesser of two evils.
    - ipunchghosts 11y ago
      Your usecase doesnt seem like HDF5 would be a good fit. Yes, I do know about docx and zip. I'm happy with what matlab has done. People send me .mat files and I happily process them in python. And my plots usually look much nicer also. :)
- deleted 11y ago
  [deleted]
- 11y ago

nippoo 11y ago

The author (my colleague, and probably the most talented developer I know) isn't replacing HDF5 with a 'proprietary binary format': in fact, the transition is as simple as replacing "HDF5 group" with "folder in a filesystem", "HDF5 dataset" with "binary file on the filesystem" (ie you store each array item sequentially on disk, exactly as HDF5 or any other format will store it, which you can memmap trivially with any programming language), and "HDF5 attribute" with "JSON / XML / INI / text file". "When in the field and the system shits the bed", to quote you... you can just open the dataset in Windows Explorer. Or Mac OS Finder. Or Nautilus. Or using 'cd' and 'ls' in Linux. Want to look at an array? Sure, open it in a hex editor or Python or, heck, FORTRAN88. You can .tar up your folder (or subfolder, or any arbitrary subset of the dataset) and send it to someone with no knowledge of the format whatsoever, and they'll be able to make sense of it in minutes. This isn't anything remotely complex - it's just using the filesystem rather than creating a filesystem-within-a-filesystem. Want to keep track of changes in your massive dataset? Sure, just back it up on Mac OS X Time Machine, or a simple rsync script, or even Git LFS. Most researchers aren't computer scientists; they know how to use Dropbox and Notepad++ and open text files, and they don't want to have to install a Java-based HDF5View when they could just use Windows Explorer. It's not even that HDF5 is that bad, it's just that filesystems are, in many respects, so much better. (If it wasn't clear from the article, we're not just misinformed - we're making this call after having developed an entire software suite around HDF5, spent about two years of firefighting HDF5 issues and wasted days of development time (so many horror stories) - this is actual feedback from several dozen users, thousands of datasets, and petabytes of data.)

ipunchghosts 11y ago
Filesystems are the worst! Try telling a customer to tar up a directory and send it to you -- things get lost so easily! Most of our customers dont even know what "tar" means. I think you are asking for trouble going with a filesystem as a storage mechanism. HDFfView is not the only viewer in town. There are several viewers. I've used HDF5 weekly for the last 5 years and so have my associates and its been wonderful. MATLAB even uses it to store its .mat files these days. I think you are misinformed and Im sticking to my story !
- nippoo 11y ago
  It depends who your users are and what needs they have. Our users are mostly MATLAB/Python users, pretty tech-savvy, and being able to edit individual bits of the dataset with other applications or write their own code to analyse them is an often-used feature. It's rare we ever need the whole dataset - in fact, it's really great to be able to say to the user "don't send us your 100GB dataset: just go to the "acquisition" subfolder and send me the 10MB file called "oscilloscope.dat"". With HDF5 this is difficult enough that it's almost always easier to send 99.9% of useless data (i.e. the whole file) when all you want is a single array within it. If your users will rarely need to do this, you could just store the entire folder hierarchy in a .zip and access it using standard tools that most programming languages have. It's worth noting that the new Microsoft Office formats do exactly this - in their case, a bunch of XML files inside a .ZIP. (Rename a .docx to .zip and you'll see!). MATLAB has moved from their own custom binary format to HDF5, which is the lesser of two evils.
  - ipunchghosts 11y ago
    Your usecase doesnt seem like HDF5 would be a good fit. Yes, I do know about docx and zip. I'm happy with what matlab has done. People send me .mat files and I happily process them in python. And my plots usually look much nicer also. :)
deleted 11y ago
[deleted]
11y ago