6 ms·

HDF5eis: Storage IO solution for big multidimensional time series sensor data

73 points by teleforce 3y ago

foobarbecue 3y ago
So the 5 is an S, and it's seismic? Nifty.
cozzyd 3y ago
wonder how it compares with ROOT or Parquet...
- fbdab103 3y ago
  I think only ROOT would be comparable. I, perhaps naively, think of Parquet as just a binary csv/table format. A modern alternative would be Zarr[0] which was frustrated with the complexity of HDF5 and wanted something that would be more amenable to working with S3 storage. Never worked with it, but the ideas are laudable. Then again, I was never all that frustrated with HDF5. I wish the exploratory tooling was better, but that is about the end of my complaints. [0] https://zarr.readthedocs.io/en/stable/index.html https://zarr.readthedocs.io/en/stable/index.html
  - prpl 3y ago
    ROOT is most comparable. Iceberg+Parquet/ORC/Avro is comparable to (and with better benefits) to some of the multi-file catalog-like features of HDF5. There’s some cases where data really needs to be structured, especially closer to instruments producing data. The thing is - some or all of that data is going to end up in a database or a dataframe sooner or later
    - dendrite9 3y ago
      I am involved in a project producing say 100 channels of 32 bit measurements that has to be roughly aligned with a series of camera images, or a video stream. My work is unconnected to the data storage question but I've been thinking about it as an interesting question I don't know enough about. I've thought about trying to implement something on a small scale as learning experience in my spare time, probably with limits on the total storage/duration. HDF5 looked like an interesting option but I've heard about issues. Do you have any suggestions for tools to look at or reading that might help push me toward doing (on not) such a project?
      - dguest 3y ago
        It depends on how much data you have and what the rest of the stack looks like. If you have O(GB) of data that needs to be accessible in one python session you could just try out h5py: https://pypi.org/project/h5py/ https://pypi.org/project/h5py/ If you're using another language there might be a more appropriate high level library.
        dendrite9 3y ago
        Yeah I guess I'd limit myself to 1-10GB just for sanity and probably use python. This isn't for customers or anything, just something that both caught my attention as a project and something I need to learn a fair amount to work on. I don't know enough about managing the data and need to figure out where to start. All the measurements need context from times before and after to be useful and I feel like I'm missing or misunderstanding some basic ideas. Thanks
  - _Wintermute 3y ago
    Zarr is gaining quite a bit of traction in bioimaging, it makes a lot of sense compared to proprietary file formats or millions of .tiff files which are both common scenarios.
    - fbdab103 3y ago
      Maybe I will revisit. I defer to being conservative when it comes to adopting data storage formats. I know I will be able to read HDF5 10 years from now. Zarr does not give me that same safe feeling yet.
- dguest 3y ago
  In my experience HDF5 is like a simplified version of ROOT: doesn't provide a full data analysis suite, graphics library, or embedded compiler. It won't assume that everything in your data maps to a C++ type, or that every C++ type should be serializable as data. It just does data, for better or worse. It also has a spec, which can be useful.
  - cozzyd 3y ago
    I'm familiar with HDF5, but I wonder how the performance of this solution built on top of HDF5 compares to ROOT (usually HDF5 does not compare favorably, but that's probably mostly due to not storing data in a chunked columnar way, which should provide similar benefits to ROOT).
    - dguest 3y ago
      The comparisons I've seen have generally been comparing ROOT on a problem it was optimized for to HDF5 (or whatever) out of the box on the same problem. There are a half dozen knobs to turn in any of these libraries, if you don't turn them I'd be skeptical of any general conclusions.
Chairesxs 3y ago
[dead]
29athrowaway 3y ago
Before moving to HDF5: https://cyrille.rossant.net/moving-away-hdf5/ https://cyrille.rossant.net/moving-away-hdf5/
- cjbgkagh 3y ago
  I was wondering if anyone else had this problem, I found HDF5 to be a nightmare to work with and is on my short list of technologies I refuse to work with anymore.
  - dguest 3y ago
    What do you recommend for binary structured data? (assuming that SQL-like isn't appropriate)
    - cjbgkagh 3y ago
      Protobuf is my go-to and I have had no issues with it. If I had to do larger memory mapped files I guess flatbuffers would be the way to go. Nympy has a reasonably sensible format, but I haven’t tried to do anything tricky with it.
      - dagw 3y ago
        Protobuf has caused me nothing but trouble once I have even moderately large data sets. If what you are storing is large matrices or rasters, honestly nothing beats HDF5.
        cjbgkagh 3y ago
        I’m old school, from the Managing Gigabytes by Ian H Witton era so I have ways of tailoring the data and protobuf schema to suit my application. Large matrices are stored as contiguous chunks, similar to how tensorflow stores tensors in graphs. I work with TBs of data in GB chunks. I used to work with PBs of data but that was a long time ago. Derialization of protobuf messages is so fast that it hasn’t been an issue and I can use it from any language with protobuf support.
teleforce 3y ago
This data format can be used for Big Data Seismology: https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2021RG000769 https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2021...
teleforce 3y ago
Full text download: https://www.researchgate.net/publication/367373175_HDF5eis_A_storage_and_IO_solution_for_big_multidimensional_time_series_data_from_environmental_sensors https://www.researchgate.net/publication/367373175_HDF5eis_A...
- neilz 3y ago
  https://www.researchgate.net/publication/43170035_STUDI_WILLINGNESS_TO_PAY_WTP_DAN_ABILITY_TO_PAY_ATP_RUAS_JALAN_TOL_DI_LUAR_PULAU_JAWA https://www.researchgate.net/publication/43170035_STUDI_WILL...
xarope 3y ago
At the risk of shoehorning SQLite into this as a solution, would it? Does SQLite handle this type of data (big multidimensional time series sensor data) well?
- jofer 3y ago
  No. The issue is that you quickly have trillions of rows. You don't need a relational db setup or fancy non-relationary db solutions. Both are well known anti-patterns. One row per sample falls over quickly for regularly sampled data. What does work for large regularly sampled datasets is array storage solutions like hdf, or image formats, etc etc. It's the same reason and same approach for your vacation photos. You wouldn't put each pixel in the db, but you might very well have a db with a row for each photo. Use a db for the non-regular portions (e.g. an index of datasets) and array storage for the actual data itself.
- osigurdson 3y ago
  Relational DBs aren't great for timeseries data which is why solutions like TimescaleDB, InfluxDB and other time series databases exist. Of course, it can work perfectly fine if the you don't have too much data. The fundamental problem with time series data is often that the insert pattern is the exact opposite of typical retrieval patterns. For example, you may insert 1000s of properties at once (most efficiently stored as interleaved data), yet typical access patterns involve obtaining a single property over many timestamps. Of course, it can work fine despite the inefficiency. If you have a few 100 sensors and are sampling them every minute, storing data for a month, it would likely be fine. If you plan on storing audio samples as records in a DB (i.e. one record per sample), it will fail. As another commenter mentions, HDF5 doesn't really help much of this. It is an unnecessary rabbit hole to some extent. It mostly just a glitchy b-tree implementation with no WAL log + some compression algorithms.
  - zX41ZdbW 3y ago
    It's not correct to say that relational DBMS are not suitable for time series. ClickHouse is a relational DBMS, and it works for time series better than specialized systems (like TimescaleDB or InfluxDB). You can quickly get trillions of records for time series, but it is not a problem. For example, see https://www.youtube.com/watch?v=JlcI2Vfz_uk https://www.youtube.com/watch?v=JlcI2Vfz_uk If you try TimescaleDB in this scenario, it will barely be crawling. PS. I am developer of ClickHouse, and I use it for these scenarios every time.
nicobrevin 3y ago
It would be nice if the hdf5 people could add better support for concurrency, a la https://www.hdfgroup.org/2020/11/webinar-enabling-multithreading-concurrency-in-hdf5-community-discussion https://www.hdfgroup.org/2020/11/webinar-enabling-multithrea... I can't imagine there'll be a queue of people wanting to implement that for fun on their weekend
- mgaunard 3y ago
  Mostly sounds like a bad idea. HDF5 is on the way out anyway.
  - klysm 3y ago
    What’s the replacement?
    - packetlost 3y ago
      HDF5 does so much there's really not a single replacement. The most prevalent is probably JSON from what I've seen, but SQLite, (C/T)SVs are also fairly popular. There's a ton of application-specific serialization formats too.
      - jjoonathan 3y ago
        None of those (JSON, SQLite, CSV) are remotely appropriate for addressing HDF5's niche of storing large blocks of numbers in a manner that can approach their native space / perf characteristics.
        dagw 3y ago
        Yea, HDF5 is great at what it is great at. The problems however come when people try to use it as a general data storage format and pseudo database.
        jjoonathan 3y ago

73 points by teleforce 3y ago

foobarbecue 3y ago

So the 5 is an S, and it's seismic? Nifty.

cozzyd 3y ago

wonder how it compares with ROOT or Parquet...

fbdab103 3y ago
I think only ROOT would be comparable. I, perhaps naively, think of Parquet as just a binary csv/table format. A modern alternative would be Zarr[0] which was frustrated with the complexity of HDF5 and wanted something that would be more amenable to working with S3 storage. Never worked with it, but the ideas are laudable. Then again, I was never all that frustrated with HDF5. I wish the exploratory tooling was better, but that is about the end of my complaints. [0] https://zarr.readthedocs.io/en/stable/index.html https://zarr.readthedocs.io/en/stable/index.html
- prpl 3y ago
  ROOT is most comparable. Iceberg+Parquet/ORC/Avro is comparable to (and with better benefits) to some of the multi-file catalog-like features of HDF5. There’s some cases where data really needs to be structured, especially closer to instruments producing data. The thing is - some or all of that data is going to end up in a database or a dataframe sooner or later
  - dendrite9 3y ago
    I am involved in a project producing say 100 channels of 32 bit measurements that has to be roughly aligned with a series of camera images, or a video stream. My work is unconnected to the data storage question but I've been thinking about it as an interesting question I don't know enough about. I've thought about trying to implement something on a small scale as learning experience in my spare time, probably with limits on the total storage/duration. HDF5 looked like an interesting option but I've heard about issues. Do you have any suggestions for tools to look at or reading that might help push me toward doing (on not) such a project?
    - dguest 3y ago
      It depends on how much data you have and what the rest of the stack looks like. If you have O(GB) of data that needs to be accessible in one python session you could just try out h5py: https://pypi.org/project/h5py/ https://pypi.org/project/h5py/ If you're using another language there might be a more appropriate high level library.
      - dendrite9 3y ago
        Yeah I guess I'd limit myself to 1-10GB just for sanity and probably use python. This isn't for customers or anything, just something that both caught my attention as a project and something I need to learn a fair amount to work on. I don't know enough about managing the data and need to figure out where to start. All the measurements need context from times before and after to be useful and I feel like I'm missing or misunderstanding some basic ideas. Thanks
- _Wintermute 3y ago
  Zarr is gaining quite a bit of traction in bioimaging, it makes a lot of sense compared to proprietary file formats or millions of .tiff files which are both common scenarios.
  - fbdab103 3y ago
    Maybe I will revisit. I defer to being conservative when it comes to adopting data storage formats. I know I will be able to read HDF5 10 years from now. Zarr does not give me that same safe feeling yet.
dguest 3y ago
In my experience HDF5 is like a simplified version of ROOT: doesn't provide a full data analysis suite, graphics library, or embedded compiler. It won't assume that everything in your data maps to a C++ type, or that every C++ type should be serializable as data. It just does data, for better or worse. It also has a spec, which can be useful.
- cozzyd 3y ago
  I'm familiar with HDF5, but I wonder how the performance of this solution built on top of HDF5 compares to ROOT (usually HDF5 does not compare favorably, but that's probably mostly due to not storing data in a chunked columnar way, which should provide similar benefits to ROOT).
  - dguest 3y ago
    The comparisons I've seen have generally been comparing ROOT on a problem it was optimized for to HDF5 (or whatever) out of the box on the same problem. There are a half dozen knobs to turn in any of these libraries, if you don't turn them I'd be skeptical of any general conclusions.

Chairesxs 3y ago

[dead]

29athrowaway 3y ago

Before moving to HDF5: https://cyrille.rossant.net/moving-away-hdf5/ https://cyrille.rossant.net/moving-away-hdf5/

cjbgkagh 3y ago
I was wondering if anyone else had this problem, I found HDF5 to be a nightmare to work with and is on my short list of technologies I refuse to work with anymore.
- dguest 3y ago
  What do you recommend for binary structured data? (assuming that SQL-like isn't appropriate)
  - cjbgkagh 3y ago
    Protobuf is my go-to and I have had no issues with it. If I had to do larger memory mapped files I guess flatbuffers would be the way to go. Nympy has a reasonably sensible format, but I haven’t tried to do anything tricky with it.
    - dagw 3y ago
      Protobuf has caused me nothing but trouble once I have even moderately large data sets. If what you are storing is large matrices or rasters, honestly nothing beats HDF5.
      - cjbgkagh 3y ago
        I’m old school, from the Managing Gigabytes by Ian H Witton era so I have ways of tailoring the data and protobuf schema to suit my application. Large matrices are stored as contiguous chunks, similar to how tensorflow stores tensors in graphs. I work with TBs of data in GB chunks. I used to work with PBs of data but that was a long time ago. Derialization of protobuf messages is so fast that it hasn’t been an issue and I can use it from any language with protobuf support.

teleforce 3y ago

This data format can be used for Big Data Seismology: https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2021RG000769 https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2021...

teleforce 3y ago

Full text download: https://www.researchgate.net/publication/367373175_HDF5eis_A_storage_and_IO_solution_for_big_multidimensional_time_series_data_from_environmental_sensors https://www.researchgate.net/publication/367373175_HDF5eis_A...

neilz 3y ago
https://www.researchgate.net/publication/43170035_STUDI_WILLINGNESS_TO_PAY_WTP_DAN_ABILITY_TO_PAY_ATP_RUAS_JALAN_TOL_DI_LUAR_PULAU_JAWA https://www.researchgate.net/publication/43170035_STUDI_WILL...

xarope 3y ago

At the risk of shoehorning SQLite into this as a solution, would it? Does SQLite handle this type of data (big multidimensional time series sensor data) well?

jofer 3y ago
No. The issue is that you quickly have trillions of rows. You don't need a relational db setup or fancy non-relationary db solutions. Both are well known anti-patterns. One row per sample falls over quickly for regularly sampled data. What does work for large regularly sampled datasets is array storage solutions like hdf, or image formats, etc etc. It's the same reason and same approach for your vacation photos. You wouldn't put each pixel in the db, but you might very well have a db with a row for each photo. Use a db for the non-regular portions (e.g. an index of datasets) and array storage for the actual data itself.
osigurdson 3y ago
Relational DBs aren't great for timeseries data which is why solutions like TimescaleDB, InfluxDB and other time series databases exist. Of course, it can work perfectly fine if the you don't have too much data. The fundamental problem with time series data is often that the insert pattern is the exact opposite of typical retrieval patterns. For example, you may insert 1000s of properties at once (most efficiently stored as interleaved data), yet typical access patterns involve obtaining a single property over many timestamps. Of course, it can work fine despite the inefficiency. If you have a few 100 sensors and are sampling them every minute, storing data for a month, it would likely be fine. If you plan on storing audio samples as records in a DB (i.e. one record per sample), it will fail. As another commenter mentions, HDF5 doesn't really help much of this. It is an unnecessary rabbit hole to some extent. It mostly just a glitchy b-tree implementation with no WAL log + some compression algorithms.
- zX41ZdbW 3y ago
  It's not correct to say that relational DBMS are not suitable for time series. ClickHouse is a relational DBMS, and it works for time series better than specialized systems (like TimescaleDB or InfluxDB). You can quickly get trillions of records for time series, but it is not a problem. For example, see https://www.youtube.com/watch?v=JlcI2Vfz_uk https://www.youtube.com/watch?v=JlcI2Vfz_uk If you try TimescaleDB in this scenario, it will barely be crawling. PS. I am developer of ClickHouse, and I use it for these scenarios every time.

nicobrevin 3y ago

It would be nice if the hdf5 people could add better support for concurrency, a la https://www.hdfgroup.org/2020/11/webinar-enabling-multithreading-concurrency-in-hdf5-community-discussion https://www.hdfgroup.org/2020/11/webinar-enabling-multithrea... I can't imagine there'll be a queue of people wanting to implement that for fun on their weekend

mgaunard 3y ago
Mostly sounds like a bad idea. HDF5 is on the way out anyway.
- klysm 3y ago
  What’s the replacement?
  - packetlost 3y ago
    HDF5 does so much there's really not a single replacement. The most prevalent is probably JSON from what I've seen, but SQLite, (C/T)SVs are also fairly popular. There's a ton of application-specific serialization formats too.
    - jjoonathan 3y ago
      None of those (JSON, SQLite, CSV) are remotely appropriate for addressing HDF5's niche of storing large blocks of numbers in a manner that can approach their native space / perf characteristics.
      - dagw 3y ago
        Yea, HDF5 is great at what it is great at. The problems however come when people try to use it as a general data storage format and pseudo database.
        jjoonathan 3y ago