3 ms·

I love HDF5! It is great for large datasets, and easy to use. It's great for one-time research projects when you need a quick solution. However, there is a lot

I love HDF5! It is great for large datasets, and easy to use. It's great for one-time research projects when you need a quick solution. However, there is a lot of room for improvement, which is why I understand the authors decision

ktpsns 7y ago
I think HDF5 really makes a good choice of usability/performance/practicability for typical user groups at laboratories or (small?) institutes. There are flaws and drawbacks which one will evventually notice. For instance, I developed (yet) a(nother) HDF5 Reader/Writer for a certain fluid dynamic code and I was quite disturbed about the way HDF5 stores meta data (i.e. attributes). There is no compression -- compression only works on actual data sets (payload). HDF5 performs bad (in terms of storage size) for many (thousands) of small data sets. This is an edge case the designers probably did not have in mind. There will be hardly anything that beats a zipped tarball containing similar information. According to the blog post, HDF5 was also not designed for deleting data sets (this was surprising for me). Parallelism is a thing... HDF5 supports there "something", but I never got happy with that. What is actually great about HDF5 is: If you are an average 2019 data scientist (not neccessarily doing HPC) and work with a decent number of slightly larger CSV files and use the defacto-standard Python-Numpy stack, making the move to HDF5 is a matter of half an hour! The h5py api is a joy to work with, and you will notice the speedup of not having to wait for long-running numpy.genfromtxt() calls.
- Enginerrrd 7y ago
  Yeah HDF5 resulted in some amazing speedups for some things I was doing with decently large data sets. It's very practical for standard use cases. The authors are really using and abusing HDF5 and based on some of the tricks they were using to eek out extra performance by going off reservation so to speak, I'm not surprised they ran into data corruption issues. That's an issue I've never had with HDF5.

ktpsns 7y ago

I think HDF5 really makes a good choice of usability/performance/practicability for typical user groups at laboratories or (small?) institutes. There are flaws and drawbacks which one will evventually notice. For instance, I developed (yet) a(nother) HDF5 Reader/Writer for a certain fluid dynamic code and I was quite disturbed about the way HDF5 stores meta data (i.e. attributes). There is no compression -- compression only works on actual data sets (payload). HDF5 performs bad (in terms of storage size) for many (thousands) of small data sets. This is an edge case the designers probably did not have in mind. There will be hardly anything that beats a zipped tarball containing similar information. According to the blog post, HDF5 was also not designed for deleting data sets (this was surprising for me). Parallelism is a thing... HDF5 supports there "something", but I never got happy with that. What is actually great about HDF5 is: If you are an average 2019 data scientist (not neccessarily doing HPC) and work with a decent number of slightly larger CSV files and use the defacto-standard Python-Numpy stack, making the move to HDF5 is a matter of half an hour! The h5py api is a joy to work with, and you will notice the speedup of not having to wait for long-running numpy.genfromtxt() calls.

Enginerrrd 7y ago
Yeah HDF5 resulted in some amazing speedups for some things I was doing with decently large data sets. It's very practical for standard use cases. The authors are really using and abusing HDF5 and based on some of the tricks they were using to eek out extra performance by going off reservation so to speak, I'm not surprised they ran into data corruption issues. That's an issue I've never had with HDF5.