3 ms·
A common pattern (that my scientific software used) was this: an initial file is created from the raw data pulled from the sequencer. After sequencing, you cou
by x0x0 11y ago
A common pattern (that my scientific software used) was this: an initial file is created from the raw data pulled from the sequencer. After sequencing, you could run all sorts of analyses. Sometimes the analyses themselves, and sometimes intermediate results, where very slow to compute and hence cached in the file.
I think it's reasonable to be very upset if you have a container file and adding new named chunks to the file has the possibility of causing the old data to become unreadable. It's fair in a crash before the file was saved that new chunks might be bad, but old chunks should be fine.
- skynetv2 11y agoGood example. definitely a problem. but that limitation exists now, so the programer would to work around it. hopefully journalling support will appear soon.
- ipunchghosts 11y agoAgreed, HDF could benefit from journaling. I have to ask though, why not just make a cached file of your data and onces its done, integrate into the final HDF file?
- ipunchghosts 11y agoThis is true, but anything can happen while futzing with a file. Our method is to make a copy of the file with a .tmp extension, make your mods, then rename the file and delete the old one. HDF5 was never intended be be a container used to stream data into -- it was meant for sharing data.