3 ms·
Mostly sounds like a bad idea. HDF5 is on the way out anyway.
by mgaunard 3y ago
Mostly sounds like a bad idea.
HDF5 is on the way out anyway.
- klysm 3y agoWhat’s the replacement?
- packetlost 3y agoHDF5 does so much there's really not a single replacement. The most prevalent is probably JSON from what I've seen, but SQLite, (C/T)SVs are also fairly popular. There's a ton of application-specific serialization formats too.
- jjoonathan 3y agoNone of those (JSON, SQLite, CSV) are remotely appropriate for addressing HDF5's niche of storing large blocks of numbers in a manner that can approach their native space / perf characteristics.
- dagw 3y agoYea, HDF5 is great at what it is great at. The problems however come when people try to use it as a general data storage format and pseudo database.
- jjoonathan 3y agoRight, it does get abused, but I don't see anyone coming for its niche and I don't see its niche evaporating.
- chaxor 3y agoLol yeah, I always reach for paperback books now since they're a great modern replacement for my Kindle. On a more serious note, DuckDB is actually a pretty fantastic replacement for SQLite for some data analytics tasks. Hdf5 is still great for numerical data in this space though. I would welcome other options like the SQLite/DuckDB scenario for matrix data.
- petschge 3y agoIf there is any replacement in the HPC world (and I don't really see indications for that), it's Adios2.
- chaxor 3y agoH5AD :D
- mgaunard 3y agoArrow.
- CreRecombinase 3y agoIt's absolutely not "on the way out". Lots of very slow moving, very deep pocketed organizations working on very long time horizons are heavily invested in HDF5. The same can't be said for any of the flavor-of-the-week ML "cloud-native" file formats.