4 ms·
I'd be curious how this handles all the posix cases not well suited to object storage. Renaming a folder than has a million files/folders inside is a single op
by hemancuso 8y ago
I'd be curious how this handles all the posix cases not well suited to object storage.
Renaming a folder than has a million files/folders inside is a single operation in SFTP, but 2 million operations on S3.
Does it handle writing at arbitrary offsets within a file? Does it download the file first then let you start writing?
What about just writing a few bytes at the beginning of a large existing file and then closing your SFTP handle?
How about 2 users accessing same file via SFTP at the same time?
- NathanKP 8y agoI don't work on the S3 team so I can't answer all these questions but relative to the "2 million operations on S3" question I can point out that S3 now has batch operation support: https://aws.amazon.com/about-aws/whats-new/2018/11/s3-batch-operations/ https://aws.amazon.com/about-aws/whats-new/2018/11/s3-batch-...
- burtonator 8y agoI think the point being that when a filesystem the mv is atomic and just updates an inode but on S3 those operations can take place on thousand of different machines.
- icebraining 8y agoSFTP is not a filesystem, though; to get a rename to be atomic, you must pass SSH_FXF_RENAME_ATOMIC, and the server can return SSH_FX_OP_UNSUPPORTED.
- hemancuso 8y agoNearly any SFTP client would assume a rename would be a near-instant operation on the server, and would probably fall-over if it took an hour.
- consultutah 8y agoThis is something that AWS needs to change. I work with millions of large files and have to keep a very large (PBs) local storage array just to make sure that things are right before uploading to S3 so that I don’t have to pay and wait for arch changes like this.