4 ms·

It would be nice if the hdf5 people could add better support for concurrency, a la https://www.hdfgroup.org/2020/11/webinar-enabling-multithreading-concurrency-

It would be nice if the hdf5 people could add better support for concurrency, a la https://www.hdfgroup.org/2020/11/webinar-enabling-multithreading-concurrency-in-hdf5-community-discussion https://www.hdfgroup.org/2020/11/webinar-enabling-multithrea... I can't imagine there'll be a queue of people wanting to implement that for fun on their weekend

mgaunard 3y ago
Mostly sounds like a bad idea. HDF5 is on the way out anyway.
- klysm 3y ago
  What’s the replacement?
  - packetlost 3y ago
    HDF5 does so much there's really not a single replacement. The most prevalent is probably JSON from what I've seen, but SQLite, (C/T)SVs are also fairly popular. There's a ton of application-specific serialization formats too.
    - jjoonathan 3y ago
      None of those (JSON, SQLite, CSV) are remotely appropriate for addressing HDF5's niche of storing large blocks of numbers in a manner that can approach their native space / perf characteristics.
      - dagw 3y ago
        Yea, HDF5 is great at what it is great at. The problems however come when people try to use it as a general data storage format and pseudo database.
        jjoonathan 3y ago
        Right, it does get abused, but I don't see anyone coming for its niche and I don't see its niche evaporating.
    - chaxor 3y ago
      Lol yeah, I always reach for paperback books now since they're a great modern replacement for my Kindle. On a more serious note, DuckDB is actually a pretty fantastic replacement for SQLite for some data analytics tasks. Hdf5 is still great for numerical data in this space though. I would welcome other options like the SQLite/DuckDB scenario for matrix data.
  - petschge 3y ago
    If there is any replacement in the HPC world (and I don't really see indications for that), it's Adios2.
  - chaxor 3y ago
    H5AD :D
  - mgaunard 3y ago
    Arrow.
- CreRecombinase 3y ago
  It's absolutely not "on the way out". Lots of very slow moving, very deep pocketed organizations working on very long time horizons are heavily invested in HDF5. The same can't be said for any of the flavor-of-the-week ML "cloud-native" file formats.
cozzyd 3y ago
There is an MPI version of HDF5 no?
- nicobrevin 3y ago
  Yeah, but no multi threading concurrency, only multi process. Not even parallel reads. There's a giant mutex around the call to read data from a dataset and my colleague's take on it was that it would be months of work to unpick all the state and make even reads thread safe.

mgaunard 3y ago

Mostly sounds like a bad idea. HDF5 is on the way out anyway.

klysm 3y ago
What’s the replacement?
- packetlost 3y ago
  HDF5 does so much there's really not a single replacement. The most prevalent is probably JSON from what I've seen, but SQLite, (C/T)SVs are also fairly popular. There's a ton of application-specific serialization formats too.
  - jjoonathan 3y ago
    None of those (JSON, SQLite, CSV) are remotely appropriate for addressing HDF5's niche of storing large blocks of numbers in a manner that can approach their native space / perf characteristics.
    - dagw 3y ago
      Yea, HDF5 is great at what it is great at. The problems however come when people try to use it as a general data storage format and pseudo database.
      - jjoonathan 3y ago
        Right, it does get abused, but I don't see anyone coming for its niche and I don't see its niche evaporating.
  - chaxor 3y ago
    Lol yeah, I always reach for paperback books now since they're a great modern replacement for my Kindle. On a more serious note, DuckDB is actually a pretty fantastic replacement for SQLite for some data analytics tasks. Hdf5 is still great for numerical data in this space though. I would welcome other options like the SQLite/DuckDB scenario for matrix data.
- petschge 3y ago
  If there is any replacement in the HPC world (and I don't really see indications for that), it's Adios2.
- chaxor 3y ago
  H5AD :D
- mgaunard 3y ago
  Arrow.
CreRecombinase 3y ago
It's absolutely not "on the way out". Lots of very slow moving, very deep pocketed organizations working on very long time horizons are heavily invested in HDF5. The same can't be said for any of the flavor-of-the-week ML "cloud-native" file formats.

cozzyd 3y ago

There is an MPI version of HDF5 no?

nicobrevin 3y ago
Yeah, but no multi threading concurrency, only multi process. Not even parallel reads. There's a giant mutex around the call to read data from a dataset and my colleague's take on it was that it would be months of work to unpick all the state and make even reads thread safe.