3 ms·
Author here! This is something I want to look into. One of the reasons to use s3fs is that the underlying `fsspec` classes already handle caching. S3File (from
by polyrand 4y ago
Author here! This is something I want to look into. One of the reasons to use s3fs is that the underlying `fsspec` classes already handle caching.
S3File (from s3fs) inherits from AbstractBufferedFile[0], which has a cache[1], implemented here[2]. I haven't read through all the code yet, but experimenting with different cache implementations will probably make the VFS faster. It will also depend on the type of queries you're executing.
[0]: https://github.com/fsspec/s3fs/blob/ad2c9b8826c75939608f5561a2a0ad33b9dafaf9/s3fs/core.py#L1830 https://github.com/fsspec/s3fs/blob/ad2c9b8826c75939608f5561...
[1]: https://github.com/fsspec/filesystem_spec/blob/2633445fc54797c79a7ac96d213bd24dfbdfcdc2/fsspec/spec.py#L1318 https://github.com/fsspec/filesystem_spec/blob/2633445fc5479...
[2]: https://github.com/fsspec/filesystem_spec/blob/2633445fc54797c79a7ac96d213bd24dfbdfcdc2/fsspec/caching.py#L118 https://github.com/fsspec/filesystem_spec/blob/2633445fc5479...
- vlovich123 4y agoDo you have any insight into whether the cache will alter consistency in the face of concurrent writes from multiple locations in a way that SQLite can't handle? I imagine this would be a very real concern for a project like this.
- polyrand 4y agoThis VFS is only for reading databases stored in S3, not writing. The README gives some alternatives for other implementations which allow writing.