4 ms·
Thank you. Memory mapping could be used for other data as well, and I have looked into zarr (even opened an issue for that https://github.com/zarr-developers/za
by maartenbreddels 8y ago
Thank you.
Memory mapping could be used for other data as well, and I have looked into zarr (even opened an issue for that https://github.com/zarr-developers/zarr/issues https://github.com/zarr-developers/zarr/issues ).
Memory mapping of contiguous data makes life much easier (for the application as well as OS), chunked data could be supported, but is more bookkeeping.
- ah- 8y agoI'll need to have a closer look later, but would vaex fit in with somewhat indexed mapped files? E.g. parquet supports column indexes now: https://issues.apache.org/jira/browse/PARQUET-1201 https://issues.apache.org/jira/browse/PARQUET-1201