4 ms·
I don't work on the S3 team so I can't answer all these questions but relative to the "2 million operations on S3" question I can point out that S3 now has batc
by NathanKP 8y ago
I don't work on the S3 team so I can't answer all these questions but relative to the "2 million operations on S3" question I can point out that S3 now has batch operation support: https://aws.amazon.com/about-aws/whats-new/2018/11/s3-batch-operations/ https://aws.amazon.com/about-aws/whats-new/2018/11/s3-batch-...
- burtonator 8y agoI think the point being that when a filesystem the mv is atomic and just updates an inode but on S3 those operations can take place on thousand of different machines.
- icebraining 8y agoSFTP is not a filesystem, though; to get a rename to be atomic, you must pass SSH_FXF_RENAME_ATOMIC, and the server can return SSH_FX_OP_UNSUPPORTED.
- hemancuso 8y agoNearly any SFTP client would assume a rename would be a near-instant operation on the server, and would probably fall-over if it took an hour.
- consultutah 8y agoThis is something that AWS needs to change. I work with millions of large files and have to keep a very large (PBs) local storage array just to make sure that things are right before uploading to S3 so that I don’t have to pay and wait for arch changes like this.