3 ms·

All excellent points, especially about cost. I've considered "faking" the append functionality by making a new file per append action, then performing a period

All excellent points, especially about cost. I've considered "faking" the append functionality by making a new file per append action, then performing a periodic compaction. Even compaction-via-combine-and-delete-old is clunky. aws s3 combine --target s3://bucket-name/output-file.txt \ s3://bucket-name/input-file-1.txt \ ... \ s3://bucket-name/input-file-n.txt I, for one, would pay extra for that.

benjiweber 11y ago
The lack of read-after-delete consistency makes this tricky. https://aws.amazon.com/s3/faqs#What_data_consistency_model_does_Amazon_S3_employ https://aws.amazon.com/s3/faqs#What_data_consistency_model_d... I've seen "eventually" consistent mean up to 24hrs in the face of problems. Several minutes seems common when versioning/bucket replication is enabled.
- mikgan 11y ago
  I can second that. Personally I'd like to see them support symbolic links so version controlling and rolling deployments of static websites becomes a little easier.
maxims 11y ago
This is actually pretty kewl. You could potentially branch the current CLI from the GitHub repo and add that functionality in. Ideally the flow would work something like the following: 1. Start a multipart object upload 2. Issue "Upload Part - Copy" requests for each part of an object ( http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html ) 3. Complete the multipart object upload Alternative flow: 1. Enable bucket versioning 2. Download the parsed S3 objects 3. Start a multipart object upload to S3 with the specified target object as the object name 4. Reupload the parsed S3 objects as parts of a single multipart object upload 5. Delete the previous parsed objects once the multipart object upload is complete (a delete marker should be added to the top of the version stack, but the previously stored version should still be available if you specify its handle/version id). Edit: changed formatting

benjiweber 11y ago

The lack of read-after-delete consistency makes this tricky. https://aws.amazon.com/s3/faqs#What_data_consistency_model_does_Amazon_S3_employ https://aws.amazon.com/s3/faqs#What_data_consistency_model_d... I've seen "eventually" consistent mean up to 24hrs in the face of problems. Several minutes seems common when versioning/bucket replication is enabled.

mikgan 11y ago
I can second that. Personally I'd like to see them support symbolic links so version controlling and rolling deployments of static websites becomes a little easier.

maxims 11y ago

This is actually pretty kewl. You could potentially branch the current CLI from the GitHub repo and add that functionality in. Ideally the flow would work something like the following: 1. Start a multipart object upload 2. Issue "Upload Part - Copy" requests for each part of an object ( http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html ) 3. Complete the multipart object upload Alternative flow: 1. Enable bucket versioning 2. Download the parsed S3 objects 3. Start a multipart object upload to S3 with the specified target object as the object name 4. Reupload the parsed S3 objects as parts of a single multipart object upload 5. Delete the previous parsed objects once the multipart object upload is complete (a delete marker should be added to the top of the version stack, but the previously stored version should still be available if you specify its handle/version id). Edit: changed formatting