5 ms·
What’s the replacement?
by klysm 3y ago
What’s the replacement?
- packetlost 3y agoHDF5 does so much there's really not a single replacement. The most prevalent is probably JSON from what I've seen, but SQLite, (C/T)SVs are also fairly popular. There's a ton of application-specific serialization formats too.
- jjoonathan 3y agoNone of those (JSON, SQLite, CSV) are remotely appropriate for addressing HDF5's niche of storing large blocks of numbers in a manner that can approach their native space / perf characteristics.
- dagw 3y agoYea, HDF5 is great at what it is great at. The problems however come when people try to use it as a general data storage format and pseudo database.
- jjoonathan 3y agoRight, it does get abused, but I don't see anyone coming for its niche and I don't see its niche evaporating.
- chaxor 3y agoLol yeah, I always reach for paperback books now since they're a great modern replacement for my Kindle. On a more serious note, DuckDB is actually a pretty fantastic replacement for SQLite for some data analytics tasks. Hdf5 is still great for numerical data in this space though. I would welcome other options like the SQLite/DuckDB scenario for matrix data.
- petschge 3y agoIf there is any replacement in the HPC world (and I don't really see indications for that), it's Adios2.
- chaxor 3y agoH5AD :D
- mgaunard 3y agoArrow.