3 ms·
People have literally used Zarr for this - at one point Gemini used Zarr for checkpointing model weights. Not sure what the current fashion in that space is tho
by tomnicholas1 9mo ago
People have literally used Zarr for this - at one point Gemini used Zarr for checkpointing model weights. Not sure what the current fashion in that space is though.
It's definitely one of many fields that see convergent evolution towards something that just looks like Zarr. In fact you can use VirtualiZarr to parse HuggingFace's "SafeTensors" format [0].
[0]: https://github.com/zarr-developers/VirtualiZarr/pull/555 https://github.com/zarr-developers/VirtualiZarr/pull/555