3 ms·
hdf5 is extremely slow however, pickle+zstd is faster and results in smaller files.
by jleahy 4y ago
hdf5 is extremely slow however, pickle+zstd is faster and results in smaller files.
- a-dub 4y agohdf5+zstd would likely be comparable. good luck loading those pickle files 5y from now.
- jleahy 4y agohdf5+zstd is not a thing (or at least not a thing that's interoperable or usable 5y from now). I just wish there was a good off-the-shelf solution, this stuff is not difficult.
- a-dub 4y agoyeah, just pointing out that there's nothing inherently wrong with hdf5 and that the gains you speak of are likely from just from the use of modern compression standards. maybe there's room for a simplified standard... or maybe just the addition of better compression to hdf5. (although they move slow for very good reason)
- hyperbovine 4y agoWhy? I have successfully loaded pkl files that are much older than that.
- welterde 4y agoI kind of feel like you are doing something wrong with HDF5, since for my use cases it's the fastest solution by far.