3 ms·
There's quite a bit of magic hiding under this. For e.g., try creating a NumPy array with different orderings, and the metadata looks the same: import asdf im
by rpep 4y ago
There's quite a bit of magic hiding under this. For e.g., try creating a NumPy array with different orderings, and the metadata looks the same:
import asdf
import numpy as np
x = np.array([[1, 2], [3, 4]], order="C")
y = np.array([[1, 2], [3, 4]], order="F")
tree = {"x": x, "y": y}
af = asdf.AsdfFile(tree)
af.write_to("example.asdf")
and you get in the metadata no distinction between the two arrays even though things like byteorder are included:
x: !core/ndarray-1.0.0
source: 2
datatype: int64
byteorder: little
shape: [2, 2]
y: !core/ndarray-1.0.0
source: 0
datatype: int64
byteorder: little
shape: [2, 2]
This makes me wonder what it's actually storing - is it actually doing something like pickling the NumPy array?
- analog31 4y agoI looked into this, because my team is considering using this format. I think it's storing array data as-is in straight binary. The type, order, and shape indicators are enough to tell how to recover the binary data.
- rpep 4y ago> I think it's storing array data as-is in straight binary. You can't infer the stride from the raw binary; a 2-D array [[1, 2], [3, 4]] in C ordering just looks like: 1,2,3,4 and in Fortran ordering it's 1,3,2,4 So there must be additional metadata stored. Maybe it's just using the .npy format internally - https://numpy.org/devdocs/reference/generated/numpy.lib.format.html https://numpy.org/devdocs/reference/generated/numpy.lib.form...
- throwaway29103 4y agobyteorder != row/column order