4 ms·

Counter argument: Binary dumps are horrible because usually the documentation that allows you to read the data is missing. Using a self-documenting format such

Counter argument: Binary dumps are horrible because usually the documentation that allows you to read the data is missing. Using a self-documenting format such as HDF5 is far superior. It will tell you of the bit are floating point numbers in single or double precision, which endianess and what the layout of the 3d array was. (No surprise that HDF was invented for the Voyager mission where they had to ensure readability of the data for half a century).

atrettel 6y ago
Your argument raises a lot of good points. I actually agree that binary does lose all of the metadata and documentation that goes with it. That is a big problem. That is why I think it is also important to include some sort of documentation like an Xdmf file [1]. That is what I use to tie everything together in my particular project. HDF5 is fine. In fact, I would have strongly preferred my colleagues using HDF5 over the proprietary format that they did end up using. But HDF5 requires an additional library. I did not want to use any external libraries in my particular project (other than MPI), so I tried to look for a solution that achieves close to what HDF5 can achieve but without requiring something as "heavy" as HDF5. I have to admit that perhaps my design choice does not work for more complex situations, but I think it is something people should consider before tying themselves down too much. [1] http://www.xdmf.org/index.php/Main_Page http://www.xdmf.org/index.php/Main_Page
- petschge 6y ago
  Having an Xdmf file alongside is nice, but the breaking changes between v2 and v3 are very annoying. And I understand the want to have few external dependencies, but at least HDF5 is straight forward to compile and available as a pre-compiled module on all supercomputers that I have ever seen.
hpcjoe 6y ago
I got into the habit of documenting each file with a file.meta that I could view later on. I did binary dumps in the past because ascii dumps (remember, 90s) were far more time/space expensive. HDF wasn't quite an option then, either HDF4, or HDF5. These days I would probably look at something like that, though, to be honest, there is always a danger of choosing something that may not be supported over the long term. This is why I generally prefer open and simple formats for everything. HDF5 is nice and open. One needs to look carefully at the total risk of using a proprietary format/system for any part of their storage. Chances are you will not be able to even read older data within a small number of decades if any of the format/system dependent technologies goes away. I've got old word processor files from the mid 80s, that I can't read. What I've written there (mostly college papers) is lost (which may be a net positive for humanity). My tarballs, and zip files though, are readable 30+ years later. That is pretty amazing. Simple, documented, and open formats. Picture a time when you can't read/open your pptx/xlsx/docx files any more. Same with data. Simple binary formats are like CSV files, but you do need to maintain metadata on their contents, and document it extensively in the code as to what you are reading/writing, why you are doing this, and how you are doing this. I think this will get more important over time as we start asking questions on how to maintain open artefact repositories for data and code. The fewer dependencies the better. And unlike the recent gene renaming snafu in biology[1], you really, never, want your tool to get in the way of the science. Either in terms of formats, or interpretation of data. [1] https://www.theverge.com/2020/8/6/21355674/human-genes-rename-microsoft-excel-misreading-dates https://www.theverge.com/2020/8/6/21355674/human-genes-renam...
iagovar 6y ago
Why not dumping into SQLite? It makes everything easy, and we will be able to use sqlite3 for a long time IMO.
- petschge 6y ago
  Because parallel IO from a lot of different MPI ranks is not supported. And filesystems tend to look unhappy when 100k processes try to open a new file at the same time.

atrettel 6y ago

Your argument raises a lot of good points. I actually agree that binary does lose all of the metadata and documentation that goes with it. That is a big problem. That is why I think it is also important to include some sort of documentation like an Xdmf file [1]. That is what I use to tie everything together in my particular project. HDF5 is fine. In fact, I would have strongly preferred my colleagues using HDF5 over the proprietary format that they did end up using. But HDF5 requires an additional library. I did not want to use any external libraries in my particular project (other than MPI), so I tried to look for a solution that achieves close to what HDF5 can achieve but without requiring something as "heavy" as HDF5. I have to admit that perhaps my design choice does not work for more complex situations, but I think it is something people should consider before tying themselves down too much. [1] http://www.xdmf.org/index.php/Main_Page http://www.xdmf.org/index.php/Main_Page

petschge 6y ago
Having an Xdmf file alongside is nice, but the breaking changes between v2 and v3 are very annoying. And I understand the want to have few external dependencies, but at least HDF5 is straight forward to compile and available as a pre-compiled module on all supercomputers that I have ever seen.

hpcjoe 6y ago

I got into the habit of documenting each file with a file.meta that I could view later on. I did binary dumps in the past because ascii dumps (remember, 90s) were far more time/space expensive. HDF wasn't quite an option then, either HDF4, or HDF5. These days I would probably look at something like that, though, to be honest, there is always a danger of choosing something that may not be supported over the long term. This is why I generally prefer open and simple formats for everything. HDF5 is nice and open. One needs to look carefully at the total risk of using a proprietary format/system for any part of their storage. Chances are you will not be able to even read older data within a small number of decades if any of the format/system dependent technologies goes away. I've got old word processor files from the mid 80s, that I can't read. What I've written there (mostly college papers) is lost (which may be a net positive for humanity). My tarballs, and zip files though, are readable 30+ years later. That is pretty amazing. Simple, documented, and open formats. Picture a time when you can't read/open your pptx/xlsx/docx files any more. Same with data. Simple binary formats are like CSV files, but you do need to maintain metadata on their contents, and document it extensively in the code as to what you are reading/writing, why you are doing this, and how you are doing this. I think this will get more important over time as we start asking questions on how to maintain open artefact repositories for data and code. The fewer dependencies the better. And unlike the recent gene renaming snafu in biology[1], you really, never, want your tool to get in the way of the science. Either in terms of formats, or interpretation of data. [1] https://www.theverge.com/2020/8/6/21355674/human-genes-rename-microsoft-excel-misreading-dates https://www.theverge.com/2020/8/6/21355674/human-genes-renam...

iagovar 6y ago

Why not dumping into SQLite? It makes everything easy, and we will be able to use sqlite3 for a long time IMO.

petschge 6y ago
Because parallel IO from a lot of different MPI ranks is not supported. And filesystems tend to look unhappy when 100k processes try to open a new file at the same time.