3 ms·

wonder how it compares with ROOT or Parquet...

by cozzyd 3y ago

fbdab103 3y ago
I think only ROOT would be comparable. I, perhaps naively, think of Parquet as just a binary csv/table format. A modern alternative would be Zarr[0] which was frustrated with the complexity of HDF5 and wanted something that would be more amenable to working with S3 storage. Never worked with it, but the ideas are laudable. Then again, I was never all that frustrated with HDF5. I wish the exploratory tooling was better, but that is about the end of my complaints. [0] https://zarr.readthedocs.io/en/stable/index.html https://zarr.readthedocs.io/en/stable/index.html
- prpl 3y ago
  ROOT is most comparable. Iceberg+Parquet/ORC/Avro is comparable to (and with better benefits) to some of the multi-file catalog-like features of HDF5. There’s some cases where data really needs to be structured, especially closer to instruments producing data. The thing is - some or all of that data is going to end up in a database or a dataframe sooner or later
  - dendrite9 3y ago
    I am involved in a project producing say 100 channels of 32 bit measurements that has to be roughly aligned with a series of camera images, or a video stream. My work is unconnected to the data storage question but I've been thinking about it as an interesting question I don't know enough about. I've thought about trying to implement something on a small scale as learning experience in my spare time, probably with limits on the total storage/duration. HDF5 looked like an interesting option but I've heard about issues. Do you have any suggestions for tools to look at or reading that might help push me toward doing (on not) such a project?
    - dguest 3y ago
      It depends on how much data you have and what the rest of the stack looks like. If you have O(GB) of data that needs to be accessible in one python session you could just try out h5py: https://pypi.org/project/h5py/ https://pypi.org/project/h5py/ If you're using another language there might be a more appropriate high level library.
      - dendrite9 3y ago
        Yeah I guess I'd limit myself to 1-10GB just for sanity and probably use python. This isn't for customers or anything, just something that both caught my attention as a project and something I need to learn a fair amount to work on. I don't know enough about managing the data and need to figure out where to start. All the measurements need context from times before and after to be useful and I feel like I'm missing or misunderstanding some basic ideas. Thanks
- _Wintermute 3y ago
  Zarr is gaining quite a bit of traction in bioimaging, it makes a lot of sense compared to proprietary file formats or millions of .tiff files which are both common scenarios.
  - fbdab103 3y ago
    Maybe I will revisit. I defer to being conservative when it comes to adopting data storage formats. I know I will be able to read HDF5 10 years from now. Zarr does not give me that same safe feeling yet.
dguest 3y ago
In my experience HDF5 is like a simplified version of ROOT: doesn't provide a full data analysis suite, graphics library, or embedded compiler. It won't assume that everything in your data maps to a C++ type, or that every C++ type should be serializable as data. It just does data, for better or worse. It also has a spec, which can be useful.
- cozzyd 3y ago
  I'm familiar with HDF5, but I wonder how the performance of this solution built on top of HDF5 compares to ROOT (usually HDF5 does not compare favorably, but that's probably mostly due to not storing data in a chunked columnar way, which should provide similar benefits to ROOT).
  - dguest 3y ago
    The comparisons I've seen have generally been comparing ROOT on a problem it was optimized for to HDF5 (or whatever) out of the box on the same problem. There are a half dozen knobs to turn in any of these libraries, if you don't turn them I'd be skeptical of any general conclusions.

wonder how it compares with ROOT or Parquet...

by cozzyd 3y ago

wonder how it compares with ROOT or Parquet...

I think only ROOT would be comparable. I, perhaps naively, think of Parquet as just a binary csv/table format. A modern alternative would be Zarr[0] which was frustrated with the complexity of HDF5 and wanted something that would be more amenable to working with S3 storage. Never worked with it, but the ideas are laudable. Then again, I was never all that frustrated with HDF5. I wish the exploratory tooling was better, but that is about the end of my complaints. [0] https://zarr.readthedocs.io/en/stable/index.html https://zarr.readthedocs.io/en/stable/index.html

prpl 3y ago
ROOT is most comparable. Iceberg+Parquet/ORC/Avro is comparable to (and with better benefits) to some of the multi-file catalog-like features of HDF5. There’s some cases where data really needs to be structured, especially closer to instruments producing data. The thing is - some or all of that data is going to end up in a database or a dataframe sooner or later
- dendrite9 3y ago
  I am involved in a project producing say 100 channels of 32 bit measurements that has to be roughly aligned with a series of camera images, or a video stream. My work is unconnected to the data storage question but I've been thinking about it as an interesting question I don't know enough about. I've thought about trying to implement something on a small scale as learning experience in my spare time, probably with limits on the total storage/duration. HDF5 looked like an interesting option but I've heard about issues. Do you have any suggestions for tools to look at or reading that might help push me toward doing (on not) such a project?
  - dguest 3y ago
    It depends on how much data you have and what the rest of the stack looks like. If you have O(GB) of data that needs to be accessible in one python session you could just try out h5py: https://pypi.org/project/h5py/ https://pypi.org/project/h5py/ If you're using another language there might be a more appropriate high level library.
    - dendrite9 3y ago
      Yeah I guess I'd limit myself to 1-10GB just for sanity and probably use python. This isn't for customers or anything, just something that both caught my attention as a project and something I need to learn a fair amount to work on. I don't know enough about managing the data and need to figure out where to start. All the measurements need context from times before and after to be useful and I feel like I'm missing or misunderstanding some basic ideas. Thanks
_Wintermute 3y ago
Zarr is gaining quite a bit of traction in bioimaging, it makes a lot of sense compared to proprietary file formats or millions of .tiff files which are both common scenarios.
- fbdab103 3y ago
  Maybe I will revisit. I defer to being conservative when it comes to adopting data storage formats. I know I will be able to read HDF5 10 years from now. Zarr does not give me that same safe feeling yet.

dguest 3y ago

In my experience HDF5 is like a simplified version of ROOT: doesn't provide a full data analysis suite, graphics library, or embedded compiler. It won't assume that everything in your data maps to a C++ type, or that every C++ type should be serializable as data. It just does data, for better or worse. It also has a spec, which can be useful.

cozzyd 3y ago
I'm familiar with HDF5, but I wonder how the performance of this solution built on top of HDF5 compares to ROOT (usually HDF5 does not compare favorably, but that's probably mostly due to not storing data in a chunked columnar way, which should provide similar benefits to ROOT).
- dguest 3y ago
  The comparisons I've seen have generally been comparing ROOT on a problem it was optimized for to HDF5 (or whatever) out of the box on the same problem. There are a half dozen knobs to turn in any of these libraries, if you don't turn them I'd be skeptical of any general conclusions.