5 ms·

I think only ROOT would be comparable. I, perhaps naively, think of Parquet as just a binary csv/table format. A modern alternative would be Zarr[0] which was

I think only ROOT would be comparable. I, perhaps naively, think of Parquet as just a binary csv/table format. A modern alternative would be Zarr[0] which was frustrated with the complexity of HDF5 and wanted something that would be more amenable to working with S3 storage. Never worked with it, but the ideas are laudable. Then again, I was never all that frustrated with HDF5. I wish the exploratory tooling was better, but that is about the end of my complaints. [0] https://zarr.readthedocs.io/en/stable/index.html https://zarr.readthedocs.io/en/stable/index.html

prpl 3y ago
ROOT is most comparable. Iceberg+Parquet/ORC/Avro is comparable to (and with better benefits) to some of the multi-file catalog-like features of HDF5. There’s some cases where data really needs to be structured, especially closer to instruments producing data. The thing is - some or all of that data is going to end up in a database or a dataframe sooner or later
- dendrite9 3y ago
  I am involved in a project producing say 100 channels of 32 bit measurements that has to be roughly aligned with a series of camera images, or a video stream. My work is unconnected to the data storage question but I've been thinking about it as an interesting question I don't know enough about. I've thought about trying to implement something on a small scale as learning experience in my spare time, probably with limits on the total storage/duration. HDF5 looked like an interesting option but I've heard about issues. Do you have any suggestions for tools to look at or reading that might help push me toward doing (on not) such a project?
  - dguest 3y ago
    It depends on how much data you have and what the rest of the stack looks like. If you have O(GB) of data that needs to be accessible in one python session you could just try out h5py: https://pypi.org/project/h5py/ https://pypi.org/project/h5py/ If you're using another language there might be a more appropriate high level library.
    - dendrite9 3y ago
      Yeah I guess I'd limit myself to 1-10GB just for sanity and probably use python. This isn't for customers or anything, just something that both caught my attention as a project and something I need to learn a fair amount to work on. I don't know enough about managing the data and need to figure out where to start. All the measurements need context from times before and after to be useful and I feel like I'm missing or misunderstanding some basic ideas. Thanks
_Wintermute 3y ago
Zarr is gaining quite a bit of traction in bioimaging, it makes a lot of sense compared to proprietary file formats or millions of .tiff files which are both common scenarios.
- fbdab103 3y ago
  Maybe I will revisit. I defer to being conservative when it comes to adopting data storage formats. I know I will be able to read HDF5 10 years from now. Zarr does not give me that same safe feeling yet.

prpl 3y ago

ROOT is most comparable. Iceberg+Parquet/ORC/Avro is comparable to (and with better benefits) to some of the multi-file catalog-like features of HDF5. There’s some cases where data really needs to be structured, especially closer to instruments producing data. The thing is - some or all of that data is going to end up in a database or a dataframe sooner or later

dendrite9 3y ago
I am involved in a project producing say 100 channels of 32 bit measurements that has to be roughly aligned with a series of camera images, or a video stream. My work is unconnected to the data storage question but I've been thinking about it as an interesting question I don't know enough about. I've thought about trying to implement something on a small scale as learning experience in my spare time, probably with limits on the total storage/duration. HDF5 looked like an interesting option but I've heard about issues. Do you have any suggestions for tools to look at or reading that might help push me toward doing (on not) such a project?
- dguest 3y ago
  It depends on how much data you have and what the rest of the stack looks like. If you have O(GB) of data that needs to be accessible in one python session you could just try out h5py: https://pypi.org/project/h5py/ https://pypi.org/project/h5py/ If you're using another language there might be a more appropriate high level library.
  - dendrite9 3y ago
    Yeah I guess I'd limit myself to 1-10GB just for sanity and probably use python. This isn't for customers or anything, just something that both caught my attention as a project and something I need to learn a fair amount to work on. I don't know enough about managing the data and need to figure out where to start. All the measurements need context from times before and after to be useful and I feel like I'm missing or misunderstanding some basic ideas. Thanks

_Wintermute 3y ago

Zarr is gaining quite a bit of traction in bioimaging, it makes a lot of sense compared to proprietary file formats or millions of .tiff files which are both common scenarios.

fbdab103 3y ago
Maybe I will revisit. I defer to being conservative when it comes to adopting data storage formats. I know I will be able to read HDF5 10 years from now. Zarr does not give me that same safe feeling yet.