3 ms·
This is cool. It would be interesting to see some stats like how many separate requests are needed for a single SELECT statement, since S3 charges per GET, even
by ks2048 4y ago
This is cool. It would be interesting to see some stats like how many separate requests are needed for a single SELECT statement, since S3 charges per GET, even on a small range, I believe. Obviously that would depend on the query and various DB settings like page_size, etc. Also wondering if this does aggressive local caching for the same reason.
- polyrand 4y agoAuthor here! This is something I want to look into. One of the reasons to use s3fs is that the underlying `fsspec` classes already handle caching. S3File (from s3fs) inherits from AbstractBufferedFile[0], which has a cache[1], implemented here[2]. I haven't read through all the code yet, but experimenting with different cache implementations will probably make the VFS faster. It will also depend on the type of queries you're executing. [0]: https://github.com/fsspec/s3fs/blob/ad2c9b8826c75939608f5561a2a0ad33b9dafaf9/s3fs/core.py#L1830 https://github.com/fsspec/s3fs/blob/ad2c9b8826c75939608f5561... [1]: https://github.com/fsspec/filesystem_spec/blob/2633445fc54797c79a7ac96d213bd24dfbdfcdc2/fsspec/spec.py#L1318 https://github.com/fsspec/filesystem_spec/blob/2633445fc5479... [2]: https://github.com/fsspec/filesystem_spec/blob/2633445fc54797c79a7ac96d213bd24dfbdfcdc2/fsspec/caching.py#L118 https://github.com/fsspec/filesystem_spec/blob/2633445fc5479...
- vlovich123 4y agoDo you have any insight into whether the cache will alter consistency in the face of concurrent writes from multiple locations in a way that SQLite can't handle? I imagine this would be a very real concern for a project like this.
- polyrand 4y agoThis VFS is only for reading databases stored in S3, not writing. The README gives some alternatives for other implementations which allow writing.