3 ms·

ROOT is most comparable. Iceberg+Parquet/ORC/Avro is comparable to (and with better benefits) to some of the multi-file catalog-like features of HDF5. There’s

by prpl 3y ago

ROOT is most comparable. Iceberg+Parquet/ORC/Avro is comparable to (and with better benefits) to some of the multi-file catalog-like features of HDF5. There’s some cases where data really needs to be structured, especially closer to instruments producing data. The thing is - some or all of that data is going to end up in a database or a dataframe sooner or later

dendrite9 3y ago
I am involved in a project producing say 100 channels of 32 bit measurements that has to be roughly aligned with a series of camera images, or a video stream. My work is unconnected to the data storage question but I've been thinking about it as an interesting question I don't know enough about. I've thought about trying to implement something on a small scale as learning experience in my spare time, probably with limits on the total storage/duration. HDF5 looked like an interesting option but I've heard about issues. Do you have any suggestions for tools to look at or reading that might help push me toward doing (on not) such a project?
- dguest 3y ago
  It depends on how much data you have and what the rest of the stack looks like. If you have O(GB) of data that needs to be accessible in one python session you could just try out h5py: https://pypi.org/project/h5py/ https://pypi.org/project/h5py/ If you're using another language there might be a more appropriate high level library.
  - dendrite9 3y ago
    Yeah I guess I'd limit myself to 1-10GB just for sanity and probably use python. This isn't for customers or anything, just something that both caught my attention as a project and something I need to learn a fair amount to work on. I don't know enough about managing the data and need to figure out where to start. All the measurements need context from times before and after to be useful and I feel like I'm missing or misunderstanding some basic ideas. Thanks

by prpl 3y ago

dendrite9 3y ago

I am involved in a project producing say 100 channels of 32 bit measurements that has to be roughly aligned with a series of camera images, or a video stream. My work is unconnected to the data storage question but I've been thinking about it as an interesting question I don't know enough about. I've thought about trying to implement something on a small scale as learning experience in my spare time, probably with limits on the total storage/duration. HDF5 looked like an interesting option but I've heard about issues. Do you have any suggestions for tools to look at or reading that might help push me toward doing (on not) such a project?

dguest 3y ago
It depends on how much data you have and what the rest of the stack looks like. If you have O(GB) of data that needs to be accessible in one python session you could just try out h5py: https://pypi.org/project/h5py/ https://pypi.org/project/h5py/ If you're using another language there might be a more appropriate high level library.
- dendrite9 3y ago
  Yeah I guess I'd limit myself to 1-10GB just for sanity and probably use python. This isn't for customers or anything, just something that both caught my attention as a project and something I need to learn a fair amount to work on. I don't know enough about managing the data and need to figure out where to start. All the measurements need context from times before and after to be useful and I feel like I'm missing or misunderstanding some basic ideas. Thanks