10 ms·
Advanced Scientific Data Format
- an1sotropy 4y agoI was hoping for some explanation of how this is envisioned to augment or coexist with HDF, a popular existing format for scientific data (but didn't see one in the docs). Did I miss it?
- lambdatronics 4y agoFrom the introduction: "On the other end of the spectrum, formats such as HDF5 and BLZ address problems with large data sets and distributed computing, but don’t really address the metadata needs of an interchange format. ASDF aims to exist in the same middle ground that made FITS so successful, by being a hybrid text and binary format: containing human editable metadata for interchange, and raw binary data that is fast to load and use. Unlike FITS, the metadata is highly structured and is designed up-front for extensibility." [0] Frankly, I feel like making the metadata of a scientific data structure human-editable is something of a mis-feature, or at best a non-feature. I use metadata in HDF5 files as a form of provenance tracking and I'd rather there be some friction to editing it. [0] https://asdf-standard.readthedocs.io/en/1.0.3/intro.html https://asdf-standard.readthedocs.io/en/1.0.3/intro.html
- chaxor 4y agoMetadata needs that, e.g., h5ad solves? I think there's quite a bit to improve on for h5 (it's very slow), but h5ad adds great ways of managing indices and meta data.
- Ultimatt 4y agoGood thing someone already invented NetCDF to address the metadata needs too...
- riedel 4y agoIn limitations it states: >While there is no hard limit on the size of the Tree, in most practical implementations it will need to be read entirely into main memory in order to interpret it, particularly to support forward references. This imposes a practical limit on its size relative to the system memory on the machine. It is not recommended to store large data sets in the tree directly, instead it should reference blocks. I would guess that HDF5 would be the better choice for large datasets. However I quite do not understand the capital 'Tree' in this sentence and what that means for practical data sets.
- derstander 4y agoIt is envisioned that this will replace HDF5, at least for astronomy — based on this 2015 paper. https://www.sciencedirect.com/science/article/pii/S2213133715000645?via%3Dihub https://www.sciencedirect.com/science/article/pii/S221313371... I’ve used HDF5 before but not ASDF so I can’t fully evaluate their points one way or the other.
- jiggawatts 4y agoWhat timing! I'm just experimenting with some bioinformatics codes, and wow... the formats are terrible. They're mostly sequential text files, some tens of gigabytes in size. Simply uncompressing a file can be a significant bottleneck, as this can inherently use only 1 CPU core. Keep in mind that these files are intended to be processed on monstrously huge 128 core and 2TB memory machines! What happens is that the system uses 0.5% of its capacity to manipulate the files while the other 99.5% is heating the data center air. I'm looking at the CPU usage metric graph now, and the machine is spending half the time at 100% load and half the time at 1% load. That's half the total capacity wasted. If you find yourself in 2022 or later designing a file format intended for bulk data and you use any of the words "stream", "serialization", or "text", stop. Rethink what you have done, and this time consider that normal machines you can buy for normal money have 256 hardware threads. Soon, 512, and then 1024 in just a few years! Everything at this scale should be split into blocks for parallel processing. Shoving text files into a .tar.gz archive is not acceptable any more. I don't care what 1970s "standard" it adheres to, that just doesn't matter any more, the hardware has moved on. I think it's high time that the industry standardised on a generic "container" format to replace legacy archive file formats. Something akin to a random-access archive file like zip, but designed so that even a single file can be decoded in parallel, even if compressed. As an example of "parallel thinking", CRC or SHA style checksums are automatically a no-go. They're inherently sequential. Instead, Merkle trees would have to be used. Compression efficiency of small files with random access could be improved by using something like zstd as the internal compression algorithm, but with a shared dictionary stored separately. This would retain the advantages of 'tar' without requiring sequential decoding of files. Etc...
- iafiaf 4y agoThe "bioinformatics formats" might be terrible, but they work. In fact, they are meant to be Excel-readable which keeps my collaborators (and me) happy. Coming from a CS /programming background, it is natural to feel the urge to "fix" the formats (<insert relevant XKCD>), until you realise that there are a libraries that easily handle serialization/parsing. Besides, "bioinformatic formats" is a meaningless word anyway. FASTQ, VCFs, BCL, AIRR-seq -- all different and it just works.
- bee_rider 4y agoI'm slightly confused about squares: !core/ndarray-1.0.0 This is supposed to be a format, but a numpy array is a python concept. So that seems like a weird mismatch. And it is just a block of numbers really, so it seems odd to make it a special python thing. I also wasn't able to see how to store more complicated data, like a symmetric sparse matrix.
- japanuspus 4y agoThe `ndarray`-concept seems to have caught on across languages for data-analysis. I have happily used `rust-ndarray`[0] for building python extensions in rust. [0]: https://docs.rs/ndarray/latest/ndarray/ https://docs.rs/ndarray/latest/ndarray/
- atemerev 4y agoI wish people stopped using YAML. It is a terrible, ambiguous, error-prone data format, particularly not suitable for data exchange purposes.
- jwr 4y agoIt's also terribly unintuitive, despite trying to be "simple and obvious". A really bad choice.
- andai 4y agoI just looked it up, looks like YAML no longer parses Norway's country code as a boolean (NO). Not sure how widespread YAML 1.2 adoption is though. https://github.com/crdoconnor/strictyaml/issues/186 https://github.com/crdoconnor/strictyaml/issues/186
- aaron695 4y agoAfter dealing with the absolute garbage that is XML I was always happy to see JSON. I agree YAML is bad and would say JSON is the best we have. What are you suggesting? I think you have to make call if you say something is terrible or a bad choice.
- zozbot234 4y agoJSON is a subset of YAML anyway. So anything that accepts YAML can deal with JSON, and you can convert one to the other.
- krapp 4y ago>JSON is a subset of YAML anyway. It isn't, really. A conforming YAML parser will not treat JSON data the same way that a JSON parser will in all cases. [0,1] The only correct way to deal with JSON is using a JSON parser. [0]https://john-millikin.com/json-is-not-a-yaml-subset https://john-millikin.com/json-is-not-a-yaml-subset [1]https://news.ycombinator.com/item?id=31406473 https://news.ycombinator.com/item?id=31406473
- mirekrusin 4y agoWhy not sqlite?
- dagss 4y agoI love SQL and use it as much as I can, but it is not a good storage format for storing gigabytes of floating point numbers that you want to do linear algebra on with routines written in Fortran or assembly, ideally without having to copy the data. And that is the exact usecase here.
- simongray 4y agoYet Another YAML-based Markup Language... or YAYML for short.
- rpep 4y agoThere's quite a bit of magic hiding under this. For e.g., try creating a NumPy array with different orderings, and the metadata looks the same: import asdf import numpy as np x = np.array([[1, 2], [3, 4]], order="C") y = np.array([[1, 2], [3, 4]], order="F") tree = {"x": x, "y": y} af = asdf.AsdfFile(tree) af.write_to("example.asdf") and you get in the metadata no distinction between the two arrays even though things like byteorder are included: x: !core/ndarray-1.0.0 source: 2 datatype: int64 byteorder: little shape: [2, 2] y: !core/ndarray-1.0.0 source: 0 datatype: int64 byteorder: little shape: [2, 2] This makes me wonder what it's actually storing - is it actually doing something like pickling the NumPy array?
- analog31 4y agoI looked into this, because my team is considering using this format. I think it's storing array data as-is in straight binary. The type, order, and shape indicators are enough to tell how to recover the binary data.
- rpep 4y ago> I think it's storing array data as-is in straight binary. You can't infer the stride from the raw binary; a 2-D array [[1, 2], [3, 4]] in C ordering just looks like: 1,2,3,4 and in Fortran ordering it's 1,3,2,4 So there must be additional metadata stored. Maybe it's just using the .npy format internally - https://numpy.org/devdocs/reference/generated/numpy.lib.format.html https://numpy.org/devdocs/reference/generated/numpy.lib.form...
- throwaway29103 4y agobyteorder != row/column order
- nness 4y agoI'm not the target audience for this, but it feels like picking a name like "asdf" means when a user has a question, quickly googling for answers is gonna be a real pain.
- billygoat 4y agoASDF is also a -different- ASDF scientific data format. There are two! Adaptable Seismic Data Format: https://asdf-definition.readthedocs.io/en/latest/ https://asdf-definition.readthedocs.io/en/latest/
- noobermin 4y agoI know ASDF is probably old by now but it would have helped if they googled first when they chose the name... for example: https://asdf.common-lisp.dev/ https://asdf.common-lisp.dev/
- nerdponx 4y agoAnd https://asdf-vm.com/ https://asdf-vm.com/
- d--b 4y agoIt seems to me that Arrow is more mature, more feature complete, more efficient and supported by many libraries. No?
- kzrdude 4y agoArrow doesn't compete with HDF so much, because the latter is a hierarchical collection of multiple datasets.
- jeffrallen 4y agoFrankly, ASDF is a downgrade from the previous Quick Words Exchanging Raw Text Yesterday spec. :)
- dmichulke 4y agoYour Undoing Is Our Progress
- msaharia 4y agoSo, this is like netCDF format?
- benreesman 4y agoI’m reminded of Erik Naggum’s epic rant about XML (I’ve misplaced the link): as the cost of compute and storage and network move relative to one another decade over decade, some things go from bad to good idea or vice versa. These days, your storage-adjacent nodes can burn compute on the fly, you have to try hard to pay for rack-adjacent 10 gig, Snappy and zstd are so fast that SREs at Google and FB turn them on or off under service owners to shave the margins. You can afford getting out of and back into a custom format for your domain, multiple times. It’s going to be mmap’d ndarrays before it hits the compute stuff, and maybe libraries could be stronger there, but you can always throw away column names and putting them back is hard.
- MikeYasnev007 4y ago
- zeitlupe 4y agoWhat is its advantage over parquet?
- zmmmmm 4y agois there anything other than a python implementation?
- watersb 4y agoIf it can become as useful as FITS, there shall be much rejoicing... Now if I could stop twitching from the five years of astronomical ontology analysis, I would feel the world has truly moved on. The timing of previous efforts was swamped by XML. Trying to maintain a clear distinction between "how" (the file format) and "what" (the content of a file in that format) can get lost in the quest to hang on to the "why" (the intent of the formatted data). I'm an optimist. There's way more experience now. Consensus around Python has helped immensely in keeping this grounded with the scientists. I have hope.
- aragilar 4y agoAre there any non-Python implementations yet? That the format is so coupled to Python is a major weakness of ASDF.
- aubanel 4y agoFor those who wonder like me what problems are solved by ASDF (why not use a big JSON file?), here's what I found. But I'd love a shorter explanation! " The following lists our key requirements for a useful format: • Human readable metadata; format suitable for archives. • Efficient support for binary data. • Implicit grouping and organization of metadata and data items. • Use of a standard format for metadata to leverage existing tools and community. • Easily extensible features, both for the general standard, and narrower needs. • Strong validation tools. • Ability to provide flexible WCS (world coordinate system) models. • References to common data or metadata without requiring copying. • Open Source, community controlled. " Source : https://aspbooks.org/a/volumes/article_details/?paper_id=40589 https://aspbooks.org/a/volumes/article_details/?paper_id=405...
- kensai 4y agoGiven it comes with specific "astronomical" problems in mind, no pun intended, can ASDF be used for other data, for example biomedical data? Would it be a good format for example for biosignals?
- osigurdson 4y agoIf a file is large enough that it cannot be read into memory in a single pass, it does beg the question if it should be in a single file in the first place. If it is that big, there may end up being multiple simultaneous writers and readers accessing various parts, depending on the use case. Now you are getting into some pretty advanced territory and will eventually need to introduce a WAL log into your file format for consistency (or crude locks). Formats like HDF5 never bothered with this leaving it up to the user to ensure that writers and readers are not accessing the file simultaneously. A directory on a file system happens to have these capability already. If you absolutely need everything in a single enormous file perhaps consider using Sqlite.
- jokteur 4y agoI know that most HPC cluster do have a limit on the number of files per user ; this pushes scientific applications to generate big hdf5 files.
- cozzyd 4y agoConsider the case where you need different parts of the same file from many processes concurrently (e.g. a lookup table). It may fit into memory but not into memory N times (yes you can use shared memory etc.). Most files are write once read many times so no need for synchronization....
- sriku 4y agoLooks like this is expected to serve as a rich file format for astronomy data. The creators of the format pick HDF5 as their best choice among other formats. So you may want to consider that. The closest discussion on the format's raison d'être is here -- "Introduction: why another data format?" - https://www.sciencedirect.com/science/article/pii/S2213133715000645?via%3Dihub#s000005 https://www.sciencedirect.com/science/article/pii/S221313371... ... which in turn points people to other papers on this, such as https://www.sciencedirect.com/science/article/pii/S2213133715000645?via%3Dihub#br000165 https://www.sciencedirect.com/science/article/pii/S221313371... . The authors say "There are a few worth considering. We will briefly review the landscape and comment on them. In short, we find significant problems with all of them. If one were to choose the best of them, it would likely be HDF5." .. and go on to list these "significant" problems as -- 1. It is an entirely binary format. 2. It is not "self documenting". 3. There is only one implementation due to the complexity. 4. HDF5 does not lend itself to supporting simpler, smaller text-based data files. 5. The HDF5 Abstract Data Model is not flexible enough to represent the structures we need to represent, notably for generalized WCS. The last seems the most significant to me. Let's see what WCS is about -- https://www.sciencedirect.com/science/article/pii/S2213133715000645?via%3Dihub#s000080 https://www.sciencedirect.com/science/article/pii/S221313371... "WCS objects consist of a sequence of coordinate frames, with a transform definition from one to the next." .. and they go on to give a "complex" example - https://www.sciencedirect.com/science/article/pii/S2213133715000645?via%3Dihub#f000030 https://www.sciencedirect.com/science/article/pii/S221313371... I'm tempted to reference https://xkcd.com/927/ https://xkcd.com/927/ , but there is nothing wrong about having a specific rich file format for astronomy data. It is just that I think folks coming to the format ought to be told that up front.
- lokimedes 4y agoFunny how, even having left particle physics a decade ago, I keep returning to CERN ROOT [0] when a dataset hits a certain size. [0] https://root.cern/primer/#file-io-and-data-analysis https://root.cern/primer/#file-io-and-data-analysis
- mahnen 4y agoThis. Python-> Uproot is all the good stuff from the ROOT data format minus the bloat https://uproot.readthedocs.io/en/latest/basic.html https://uproot.readthedocs.io/en/latest/basic.html
- saboot 4y agoUproot really kept CERN ROOT from falling into obscurity.
- cozzyd 4y agoWhile it's very nice to have uproot (and I have contributed to it!), unfortunately it isn't able to read all of the ROOT files I generate... probably because I'm not aware of its limitations. (See e.g. https://github.com/scikit-hep/uproot5/issues/586 https://github.com/scikit-hep/uproot5/issues/586 ) Of course solving the general problem of all ROOT I/O (which can involve custom serialization or schema migration code in C++) won't (and doesn't need to, for most people) work.
- cbarrick 4y agoWhat are the benefits over HDF5 / netCDF? Python already has great support for these formats with Dask and Xarray. (Think multidimensional Pandas.)
- mattewong 4y agoI'm having a hard time clearly seeing what the problems were with other approaches that ASDF is intended to solve. Are the other solutions not fast enough? If so what is the performance delta using ASDF? Are the other solutions not scalable enough? Likely not, given that ASDF mentions certain scale limitations-- but maybe those limitations are still less limiting than limitations of other solutions? Whatever the reasons for making ADFS, it would be helpful to describe in the intro "here is a problem that ADFS solves" and "here is a graph (or similar) showing how it solves it better than alternatives". Without that context, the description of Features, to my naive outsider's view, is not very valuable and I'm left still asking, "why use this instead of other solutions that many engineers already are familiar with?"
- julienchastang 4y agoBeyond what has been mentioned so far (i.e., how is this better than long-standing scientific data formats and tools, e.g., netcdf, hdf, numpy, dask, etc. ?) I would like to know if there is a cloud native data storage aspect to asdf similar to the work the zarr folks are tackling. As data sets grow increasingly large, we move computation to data instead of the other way around. Does asdf meld with that paradigm? Glancing through the docs, I did not see it mentioned, but perhaps I missed it.