4 ms·
It looks quite nice, and I will have to explore the performance comparisons with Dask more. I have recently started using Xarray for some projects, and really
by rax 8y ago
It looks quite nice, and I will have to explore the performance comparisons with Dask more.
I have recently started using Xarray for some projects, and really appreciate the usability of multidimensional labelled data. Are the memory mapping techniques used for speedup here only applicable to tabular data?
The support for Apache arrow is quite nice. Have you considered any other formats, such as Zarr?
- maartenbreddels 8y agoThank you. Memory mapping could be used for other data as well, and I have looked into zarr (even opened an issue for that https://github.com/zarr-developers/zarr/issues https://github.com/zarr-developers/zarr/issues ). Memory mapping of contiguous data makes life much easier (for the application as well as OS), chunked data could be supported, but is more bookkeeping.
- ah- 8y agoI'll need to have a closer look later, but would vaex fit in with somewhat indexed mapped files? E.g. parquet supports column indexes now: https://issues.apache.org/jira/browse/PARQUET-1201 https://issues.apache.org/jira/browse/PARQUET-1201