3 ms·

> The data has to be on the filesystem. You can’t load data from a blob store like AWS S3. You sure about this? You can mount S3 locally, and the filesystem sh

• itamarst 7y ago
  There are two ways of doing this that I know of: 1. FUSE filesystems. This means sending really slow queries to S3, and so you're much better off using compression to get better performance. 2. Syncing to filesystem with AWS DataSync. This would work, yes, but it's a S3 specific feature and not all blob storage systems have it.
  • jedbrown 7y ago
    HDF5 has some S3-native features emerging (https://www.hdfgroup.org/wp-content/uploads/2019/11/SC19-HDF5-BOF.pdf https://www.hdfgroup.org/wp-content/uploads/2019/11/SC19-HDF...), and there's also this relevant comparison reading HDF5 files as efficiently as native Zarr: https://medium.com/pangeo/cloud-performant-reading-of-netcdf4-hdf5-data-using-the-zarr-library-1a95c5c92314 https://medium.com/pangeo/cloud-performant-reading-of-netcdf...
  • khc 7y ago
    > 1. FUSE filesystems. This means sending really slow queries to S3, and so you're much better off using compression to get better performance. As author of https://github.com/kahing/goofys/ https://github.com/kahing/goofys/ I respectfully disagree :-)
    • itamarst 7y ago
      I don't mean the filesystem implementation is inherently slow, necessarily: I'm talking about bandwidth. If you get 3× compression, read times from S3 will be 3× as fast—but that means no mmap().