Over time, I've found it is quite possible to mitigate most of these issues. The default settings are not great, mostly because they default to maximize backwar

Over time, I've found it is quite possible to mitigate most of these issues. The default settings are not great, mostly because they default to maximize backwards compatibility. With a few settings you can consolidate the metadata such that the data layout is very simple. The structure of a HDF5 file can be manipulated to put data where you want it in a file. For many files of the same type, you can set the metablock size so that your datasets are always at the same offsets. You can use tools like h5ls or h5dump to describe the structure of a HDF5 file. It turns out that if you do this, you can make HDF5 cloud friendly. https://medium.com/pangeo/cloud-performant-netcdf4-hdf5-with-zarr-fsspec-and-intake-3d3a3e7cb935 https://medium.com/pangeo/cloud-performant-netcdf4-hdf5-with...