4 ms·

Before moving to HDF5: https://cyrille.rossant.net/moving-away-hdf5/ https://cyrille.rossant.net/moving-away-hdf5/

cjbgkagh 3y ago
I was wondering if anyone else had this problem, I found HDF5 to be a nightmare to work with and is on my short list of technologies I refuse to work with anymore.
- dguest 3y ago
  What do you recommend for binary structured data? (assuming that SQL-like isn't appropriate)
  - cjbgkagh 3y ago
    Protobuf is my go-to and I have had no issues with it. If I had to do larger memory mapped files I guess flatbuffers would be the way to go. Nympy has a reasonably sensible format, but I haven’t tried to do anything tricky with it.
    - dagw 3y ago
      Protobuf has caused me nothing but trouble once I have even moderately large data sets. If what you are storing is large matrices or rasters, honestly nothing beats HDF5.
      - cjbgkagh 3y ago
        I’m old school, from the Managing Gigabytes by Ian H Witton era so I have ways of tailoring the data and protobuf schema to suit my application. Large matrices are stored as contiguous chunks, similar to how tensorflow stores tensors in graphs. I work with TBs of data in GB chunks. I used to work with PBs of data but that was a long time ago. Derialization of protobuf messages is so fast that it hasn’t been an issue and I can use it from any language with protobuf support.
  - saltcured 3y ago
    I think that depends on the use case and full data lifecycle. How is data acquired and added to datasets? How is it consumed? An online, database-like product is most important if you need to coordinate concurrent activity by a number of data producers and consumers while maintaining a coherent view of the growing data. If you can break it into distinct phases with individual actors, passive serialization formats can make more sense. Adoption of bject-storage semantics would help eliminate some of the corruption/concurrency hazards mentioned in Cyrille's post. You write entire files and expose them to read-only consumers once they are valid and complete, side-stepping concurrent writer/reader scenarios. However, object-storage still has coherence problems. If you expect metadata to need rounds of editing or curation, you don't want it embedded in bulk objects, where mutation is expensive, i.e. rewriting an entirely new version of the object. It is easier with a companion file strategy, where you can regenerate smaller metadata files alongside immutable bulk data files. The object store can ensure that concurrent users are only encountering a coherent snapshot of each metadata file, i.e. before or after it was replaced. But it does not provide coherent views of multi-file changes. You avoid low-level codec failures from concurrent writes to shared files, but you may still have semantic inconsistencies when interpreting the evolving multi-object dataset. A hybrid approach would be to have some other data-management system with database-like properties to track the objects. The database stores a coherent catalog of which data objects are available. New objects are added and registered in the catalog for others to read, but you don't mutate them afterwards. The catalog should contain metadata needed to coordinate the workflows of production and consumption of objects. You could benefit from using the catalog as an authoritative store of metadata that is still being curated/refined. But you can also export snapshots of metadata into companion objects at appropriate milestones in the processing workflow. These could even be versioned to record well-defined "data releases" or provenance info related to individual processing tasks. Of course, this still leaves the question of what file format(s) to use for the individual objects in the system... one could do all of the above and still choose to use HDF5! But these individual files would be simpler, i.e. to store a single N-dimensional numerical array in one file. Microscopy projects might choose OME-TIFF for individual image stacks, or something even more mundane like a directory of individual 2D TIFF or PNG files, if that maps well to how their acquisition or processing pipeline works. Sometimes, the right file layout is just as important as choosing chunking parameters within a format like HDF5. Naive pipelines often process whole files in RAM, so you end up choosing appropriate file layouts to align to the kind of sparse or sequential access needed by data producer and consumer programs.
  - rossant 3y ago
    We're using the NPY format [1] at the International Brain Laboratory and we're pretty happy with it. It works well if you have few large arrays that you can store in different files within a folder. Another good option is Apache Parquet. [1] https://numpy.org/devdocs/reference/generated/numpy.lib.format.html https://numpy.org/devdocs/reference/generated/numpy.lib.form...
- zwaps 3y ago
  Yes. I had to save string data in a column, which is frankly not something you want to do. HDF5 is super weird about encodings, quietly introduces errors, requires a fixed character length for that column and if you want to quickly access or search through the data, you have to write a program because HDF5View (or whatever it was called) is the worst. And since you still have to manage multiple reads and writes, the only benefit is maybe speed. String data and HDF5 = No. It does have an index tho compared to Parquet, which is handy for ML type stuff.
- xmcqdpt2 3y ago
  I worked with HDF5 a bunch for physics simulation and it was extremely fragile. Performance was good on a distributed hard drive when reading and loading blocked data over many machines but files would corrupt whenever a compute node would crash. Maybe they fixed it since, that was in 2014 or so. It's a quantum file format. It works so long as you aren't looking at it too much.
  - deleted 3y ago
    [deleted]
dguest 3y ago
I was pretty happy with it. But then, I was moving from something much worse that very few people use.
tired_star_nrg 3y ago
Oh wow, sounds very similar to trying to use Microsoft Access files for production at scale. Many of the same headaches with everything in a single file
- osigurdson 3y ago
  >> everything in a single file A file is just an abstraction over a block device. HDF5 is a meta abstraction, storing multiple file abstractions within an existing container file abstraction. There is nothing conceptually wrong about this but incredible to me that anyone would have invested the time into creating an abstraction that looks exactly like its container just to avoid having to tar it.
wnkrshm 3y ago
You cannot load or write (may only have been one of those operations, strangely) a file containing non-ASCII characters with the C/C++ libraries if you target Win. If you target Linux, those work out of the box due to the file system doing the lifting. With windows we had to use the short file name (e.g. FILEN~1.EXT) as a workaround. Also, you have to watch out for what the library does when it writes to (Edit: Windows) network shares - we've seen it write NaNs where there were values in the data it should save, maybe a latency issue, maybe a configuration issue - but not something you get told about. I would like to move away from it.
- planede 3y ago
  > You cannot load or write (may only have been one of those operations, strangely) a file containing non-ASCII characters with the C/C++ libraries if you target Win. This is definitely not true, at least for C++, but I agree that it can be hassle.
markkitti 3y ago
Over time, I've found it is quite possible to mitigate most of these issues. The default settings are not great, mostly because they default to maximize backwards compatibility. With a few settings you can consolidate the metadata such that the data layout is very simple. The structure of a HDF5 file can be manipulated to put data where you want it in a file. For many files of the same type, you can set the metablock size so that your datasets are always at the same offsets. You can use tools like h5ls or h5dump to describe the structure of a HDF5 file. It turns out that if you do this, you can make HDF5 cloud friendly. https://medium.com/pangeo/cloud-performant-netcdf4-hdf5-with-zarr-fsspec-and-intake-3d3a3e7cb935 https://medium.com/pangeo/cloud-performant-netcdf4-hdf5-with...
wiz21c 3y ago
what are the alternatives to HDF5 then ? (I thought HDF5 was a de facto standard, I feel wrong now :-))
- dguest 3y ago
  It is a de facto standard: you can tell because the author of that post suggests rolling your own simpler data format as an alternative. There are indeed a lot of reasons not to use HDF5: if you have a lot of small records, are primarily storing strings, or you don't share your data with many colleagues, there are a lot of better alternatives. But if you just want to dump a few hundred GB into one massive file and have decent IO in a format people can figure out, it works pretty good.

I was wondering if anyone else had this problem, I found HDF5 to be a nightmare to work with and is on my short list of technologies I refuse to work with anymore.

dguest 3y ago
What do you recommend for binary structured data? (assuming that SQL-like isn't appropriate)
- cjbgkagh 3y ago
  Protobuf is my go-to and I have had no issues with it. If I had to do larger memory mapped files I guess flatbuffers would be the way to go. Nympy has a reasonably sensible format, but I haven’t tried to do anything tricky with it.
  - dagw 3y ago
    Protobuf has caused me nothing but trouble once I have even moderately large data sets. If what you are storing is large matrices or rasters, honestly nothing beats HDF5.
    - cjbgkagh 3y ago
      I’m old school, from the Managing Gigabytes by Ian H Witton era so I have ways of tailoring the data and protobuf schema to suit my application. Large matrices are stored as contiguous chunks, similar to how tensorflow stores tensors in graphs. I work with TBs of data in GB chunks. I used to work with PBs of data but that was a long time ago. Derialization of protobuf messages is so fast that it hasn’t been an issue and I can use it from any language with protobuf support.
- saltcured 3y ago
  I think that depends on the use case and full data lifecycle. How is data acquired and added to datasets? How is it consumed? An online, database-like product is most important if you need to coordinate concurrent activity by a number of data producers and consumers while maintaining a coherent view of the growing data. If you can break it into distinct phases with individual actors, passive serialization formats can make more sense. Adoption of bject-storage semantics would help eliminate some of the corruption/concurrency hazards mentioned in Cyrille's post. You write entire files and expose them to read-only consumers once they are valid and complete, side-stepping concurrent writer/reader scenarios. However, object-storage still has coherence problems. If you expect metadata to need rounds of editing or curation, you don't want it embedded in bulk objects, where mutation is expensive, i.e. rewriting an entirely new version of the object. It is easier with a companion file strategy, where you can regenerate smaller metadata files alongside immutable bulk data files. The object store can ensure that concurrent users are only encountering a coherent snapshot of each metadata file, i.e. before or after it was replaced. But it does not provide coherent views of multi-file changes. You avoid low-level codec failures from concurrent writes to shared files, but you may still have semantic inconsistencies when interpreting the evolving multi-object dataset. A hybrid approach would be to have some other data-management system with database-like properties to track the objects. The database stores a coherent catalog of which data objects are available. New objects are added and registered in the catalog for others to read, but you don't mutate them afterwards. The catalog should contain metadata needed to coordinate the workflows of production and consumption of objects. You could benefit from using the catalog as an authoritative store of metadata that is still being curated/refined. But you can also export snapshots of metadata into companion objects at appropriate milestones in the processing workflow. These could even be versioned to record well-defined "data releases" or provenance info related to individual processing tasks. Of course, this still leaves the question of what file format(s) to use for the individual objects in the system... one could do all of the above and still choose to use HDF5! But these individual files would be simpler, i.e. to store a single N-dimensional numerical array in one file. Microscopy projects might choose OME-TIFF for individual image stacks, or something even more mundane like a directory of individual 2D TIFF or PNG files, if that maps well to how their acquisition or processing pipeline works. Sometimes, the right file layout is just as important as choosing chunking parameters within a format like HDF5. Naive pipelines often process whole files in RAM, so you end up choosing appropriate file layouts to align to the kind of sparse or sequential access needed by data producer and consumer programs.
- rossant 3y ago
  We're using the NPY format [1] at the International Brain Laboratory and we're pretty happy with it. It works well if you have few large arrays that you can store in different files within a folder. Another good option is Apache Parquet. [1] https://numpy.org/devdocs/reference/generated/numpy.lib.format.html https://numpy.org/devdocs/reference/generated/numpy.lib.form...
zwaps 3y ago
Yes. I had to save string data in a column, which is frankly not something you want to do. HDF5 is super weird about encodings, quietly introduces errors, requires a fixed character length for that column and if you want to quickly access or search through the data, you have to write a program because HDF5View (or whatever it was called) is the worst. And since you still have to manage multiple reads and writes, the only benefit is maybe speed. String data and HDF5 = No. It does have an index tho compared to Parquet, which is handy for ML type stuff.
xmcqdpt2 3y ago
I worked with HDF5 a bunch for physics simulation and it was extremely fragile. Performance was good on a distributed hard drive when reading and loading blocked data over many machines but files would corrupt whenever a compute node would crash. Maybe they fixed it since, that was in 2014 or so. It's a quantum file format. It works so long as you aren't looking at it too much.
- deleted 3y ago
  [deleted]

dguest 3y ago

I was pretty happy with it. But then, I was moving from something much worse that very few people use.

tired_star_nrg 3y ago

Oh wow, sounds very similar to trying to use Microsoft Access files for production at scale. Many of the same headaches with everything in a single file

osigurdson 3y ago
>> everything in a single file A file is just an abstraction over a block device. HDF5 is a meta abstraction, storing multiple file abstractions within an existing container file abstraction. There is nothing conceptually wrong about this but incredible to me that anyone would have invested the time into creating an abstraction that looks exactly like its container just to avoid having to tar it.

wnkrshm 3y ago

You cannot load or write (may only have been one of those operations, strangely) a file containing non-ASCII characters with the C/C++ libraries if you target Win. If you target Linux, those work out of the box due to the file system doing the lifting. With windows we had to use the short file name (e.g. FILEN~1.EXT) as a workaround. Also, you have to watch out for what the library does when it writes to (Edit: Windows) network shares - we've seen it write NaNs where there were values in the data it should save, maybe a latency issue, maybe a configuration issue - but not something you get told about. I would like to move away from it.

planede 3y ago
> You cannot load or write (may only have been one of those operations, strangely) a file containing non-ASCII characters with the C/C++ libraries if you target Win. This is definitely not true, at least for C++, but I agree that it can be hassle.

markkitti 3y ago

Over time, I've found it is quite possible to mitigate most of these issues. The default settings are not great, mostly because they default to maximize backwards compatibility. With a few settings you can consolidate the metadata such that the data layout is very simple. The structure of a HDF5 file can be manipulated to put data where you want it in a file. For many files of the same type, you can set the metablock size so that your datasets are always at the same offsets. You can use tools like h5ls or h5dump to describe the structure of a HDF5 file. It turns out that if you do this, you can make HDF5 cloud friendly. https://medium.com/pangeo/cloud-performant-netcdf4-hdf5-with-zarr-fsspec-and-intake-3d3a3e7cb935 https://medium.com/pangeo/cloud-performant-netcdf4-hdf5-with...

wiz21c 3y ago

what are the alternatives to HDF5 then ? (I thought HDF5 was a de facto standard, I feel wrong now :-))

dguest 3y ago
It is a de facto standard: you can tell because the author of that post suggests rolling your own simpler data format as an alternative. There are indeed a lot of reasons not to use HDF5: if you have a lot of small records, are primarily storing strings, or you don't share your data with many colleagues, there are a lot of better alternatives. But if you just want to dump a few hundred GB into one massive file and have decent IO in a format people can figure out, it works pretty good.