3 ms·
There are two ways of doing this that I know of: 1. FUSE filesystems. This means sending really slow queries to S3, and so you're much better off using compres
by itamarst 7y ago
There are two ways of doing this that I know of:
1. FUSE filesystems. This means sending really slow queries to S3, and so you're much better off using compression to get better performance.
2. Syncing to filesystem with AWS DataSync. This would work, yes, but it's a S3 specific feature and not all blob storage systems have it.
- jedbrown 7y agoHDF5 has some S3-native features emerging (https://www.hdfgroup.org/wp-content/uploads/2019/11/SC19-HDF5-BOF.pdf https://www.hdfgroup.org/wp-content/uploads/2019/11/SC19-HDF...), and there's also this relevant comparison reading HDF5 files as efficiently as native Zarr: https://medium.com/pangeo/cloud-performant-reading-of-netcdf4-hdf5-data-using-the-zarr-library-1a95c5c92314 https://medium.com/pangeo/cloud-performant-reading-of-netcdf...
- khc 7y ago> 1. FUSE filesystems. This means sending really slow queries to S3, and so you're much better off using compression to get better performance. As author of https://github.com/kahing/goofys/ https://github.com/kahing/goofys/ I respectfully disagree :-)
- itamarst 7y agoI don't mean the filesystem implementation is inherently slow, necessarily: I'm talking about bandwidth. If you get 3× compression, read times from S3 will be 3× as fast—but that means no mmap().