3 ms·
If you know the exact tensor shape of your data ahead of time Zarr works well (we use it as the dataformat for our ml experiments). If you have dynamically grow
by lysecret 2y ago
If you know the exact tensor shape of your data ahead of time Zarr works well (we use it as the dataformat for our ml experiments). If you have dynamically growing data or irregular shapes zarr doesn't work as well.
- tomnicholas1 2y agoIcechunk can handle growing dimensions with ACID transactions! For irregular shapes in some cases using multiple groups + xarray.DataTree can help you, but in general yeah ragged data is hard.