3 ms·
Looks like this is expected to serve as a rich file format for astronomy data. The creators of the format pick HDF5 as their best choice among other formats. So
by sriku 4y ago
Looks like this is expected to serve as a rich file format for astronomy data. The creators of the format pick HDF5 as their best choice among other formats. So you may want to consider that.
The closest discussion on the format's raison d'être is here -- "Introduction: why another data format?" - https://www.sciencedirect.com/science/article/pii/S2213133715000645?via%3Dihub#s000005 https://www.sciencedirect.com/science/article/pii/S221313371... ... which in turn points people to other papers on this, such as https://www.sciencedirect.com/science/article/pii/S2213133715000645?via%3Dihub#br000165 https://www.sciencedirect.com/science/article/pii/S221313371... .
The authors say "There are a few worth considering. We will briefly review the landscape and comment on them. In short, we find significant problems with all of them. If one were to choose the best of them, it would likely be HDF5." .. and go on to list these "significant" problems as --
1. It is an entirely binary format.
2. It is not "self documenting".
3. There is only one implementation due to the complexity.
4. HDF5 does not lend itself to supporting simpler, smaller text-based data files.
5. The HDF5 Abstract Data Model is not flexible enough to represent the structures we need to represent, notably for generalized WCS.
The last seems the most significant to me. Let's see what WCS is about -- https://www.sciencedirect.com/science/article/pii/S2213133715000645?via%3Dihub#s000080 https://www.sciencedirect.com/science/article/pii/S221313371...
"WCS objects consist of a sequence of coordinate frames, with a transform definition from one to the next."
.. and they go on to give a "complex" example - https://www.sciencedirect.com/science/article/pii/S2213133715000645?via%3Dihub#f000030 https://www.sciencedirect.com/science/article/pii/S221313371...
I'm tempted to reference https://xkcd.com/927/ https://xkcd.com/927/ , but there is nothing wrong about having a specific rich file format for astronomy data. It is just that I think folks coming to the format ought to be told that up front.