3 ms·

S3 is not a filesystem replacement unfortunately.

by maccard 5y ago

kumarsw 5y ago
Not a filesystem, but rclone works pretty well
- maccard 5y ago
  rclone also works with Google drive, if that's all you're after.
yjftsjthsd-h 5y ago
Why not? It's a cloud service that stores files, can pretend to have directories, and for which there exist multiple utilities to mount the thing as a local file system. What's the shortcoming?
- mr_toad 5y ago
  S3 is object storage not block storage. https://www.digitalocean.com/community/tutorials/object-storage-vs-block-storage-services https://www.digitalocean.com/community/tutorials/object-stor... Object storage isn’t random access (the ability to read and write any part of a file) the way a disk drive is. This causes complications for software that expects to be able to randomly access a file like they would on a disk.
  - drdaeman 5y ago
    WebDAV is also an object storage protocol but it can do random access. Well, almost. Partial reads are supported everywhere out of box just by using the Content-Range HTTP header. Partial writes aren't standard, but some implementations e.g. SabreDAV can do those using PATCH with X-Update-Range header (and e.g. rclone supports this). WebDAV has its warts and limitations but it's a very underappreciated protocol.
- maccard 5y ago
  As someone else pointed out, it's object storage not block storage meaning things like partial reads and partial writes don't work. That may not be a problem for you, but it _is_ a problem for me (particularly when dealing with remote data). > for which there exist multiple utilities to mount the thing as a local file system Which all have their own sets of intricacies. The most obvious one being listing a directory is either impossible, costly, or as slow as the heat death of the universe. Other (far riskier) ones (from s3fs-fuse, which I have the most experience with[0] ) are the lack of guarantees about atomicity and concurrent access. [0] https://github.com/s3fs-fuse/s3fs-fuse https://github.com/s3fs-fuse/s3fs-fuse

S3 is not a filesystem replacement unfortunately.

kumarsw 5y ago

Not a filesystem, but rclone works pretty well

maccard 5y ago
rclone also works with Google drive, if that's all you're after.

yjftsjthsd-h 5y ago

Why not? It's a cloud service that stores files, can pretend to have directories, and for which there exist multiple utilities to mount the thing as a local file system. What's the shortcoming?

mr_toad 5y ago
S3 is object storage not block storage. https://www.digitalocean.com/community/tutorials/object-storage-vs-block-storage-services https://www.digitalocean.com/community/tutorials/object-stor... Object storage isn’t random access (the ability to read and write any part of a file) the way a disk drive is. This causes complications for software that expects to be able to randomly access a file like they would on a disk.
- drdaeman 5y ago
  WebDAV is also an object storage protocol but it can do random access. Well, almost. Partial reads are supported everywhere out of box just by using the Content-Range HTTP header. Partial writes aren't standard, but some implementations e.g. SabreDAV can do those using PATCH with X-Update-Range header (and e.g. rclone supports this). WebDAV has its warts and limitations but it's a very underappreciated protocol.
maccard 5y ago
As someone else pointed out, it's object storage not block storage meaning things like partial reads and partial writes don't work. That may not be a problem for you, but it _is_ a problem for me (particularly when dealing with remote data). > for which there exist multiple utilities to mount the thing as a local file system Which all have their own sets of intricacies. The most obvious one being listing a directory is either impossible, costly, or as slow as the heat death of the universe. Other (far riskier) ones (from s3fs-fuse, which I have the most experience with[0] ) are the lack of guarantees about atomicity and concurrent access. [0] https://github.com/s3fs-fuse/s3fs-fuse https://github.com/s3fs-fuse/s3fs-fuse