3 ms·

I don't think it has a traditional filesystem. It probably just writes all puts sequentially as fast as possible and stores the location and then replicates.

I don't think it has a traditional filesystem. It probably just writes all puts sequentially as fast as possible and stores the location and then replicates. The easiest way to append would be to read the object, append, and then write to a new object. If they did that internally there would be no transfer out and no revenue although they could probably charge for the internal expense. Another reason is that people would probably think that appends are no big deal and try to append continuously to multi-gigabyte files. If this is the case then it is best to let the client handle appends where costs are out in the open.

whatnotests 11y ago
All excellent points, especially about cost. I've considered "faking" the append functionality by making a new file per append action, then performing a periodic compaction. Even compaction-via-combine-and-delete-old is clunky. aws s3 combine --target s3://bucket-name/output-file.txt \ s3://bucket-name/input-file-1.txt \ ... \ s3://bucket-name/input-file-n.txt I, for one, would pay extra for that.
- benjiweber 11y ago
  The lack of read-after-delete consistency makes this tricky. https://aws.amazon.com/s3/faqs#What_data_consistency_model_does_Amazon_S3_employ https://aws.amazon.com/s3/faqs#What_data_consistency_model_d... I've seen "eventually" consistent mean up to 24hrs in the face of problems. Several minutes seems common when versioning/bucket replication is enabled.
  - mikgan 11y ago
    I can second that. Personally I'd like to see them support symbolic links so version controlling and rolling deployments of static websites becomes a little easier.
- maxims 11y ago
  This is actually pretty kewl. You could potentially branch the current CLI from the GitHub repo and add that functionality in. Ideally the flow would work something like the following: 1. Start a multipart object upload 2. Issue "Upload Part - Copy" requests for each part of an object ( http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html ) 3. Complete the multipart object upload Alternative flow: 1. Enable bucket versioning 2. Download the parsed S3 objects 3. Start a multipart object upload to S3 with the specified target object as the object name 4. Reupload the parsed S3 objects as parts of a single multipart object upload 5. Delete the previous parsed objects once the multipart object upload is complete (a delete marker should be added to the top of the version stack, but the previously stored version should still be available if you specify its handle/version id). Edit: changed formatting

whatnotests 11y ago

All excellent points, especially about cost. I've considered "faking" the append functionality by making a new file per append action, then performing a periodic compaction. Even compaction-via-combine-and-delete-old is clunky. aws s3 combine --target s3://bucket-name/output-file.txt \ s3://bucket-name/input-file-1.txt \ ... \ s3://bucket-name/input-file-n.txt I, for one, would pay extra for that.

benjiweber 11y ago
The lack of read-after-delete consistency makes this tricky. https://aws.amazon.com/s3/faqs#What_data_consistency_model_does_Amazon_S3_employ https://aws.amazon.com/s3/faqs#What_data_consistency_model_d... I've seen "eventually" consistent mean up to 24hrs in the face of problems. Several minutes seems common when versioning/bucket replication is enabled.
- mikgan 11y ago
  I can second that. Personally I'd like to see them support symbolic links so version controlling and rolling deployments of static websites becomes a little easier.
maxims 11y ago
This is actually pretty kewl. You could potentially branch the current CLI from the GitHub repo and add that functionality in. Ideally the flow would work something like the following: 1. Start a multipart object upload 2. Issue "Upload Part - Copy" requests for each part of an object ( http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html ) 3. Complete the multipart object upload Alternative flow: 1. Enable bucket versioning 2. Download the parsed S3 objects 3. Start a multipart object upload to S3 with the specified target object as the object name 4. Reupload the parsed S3 objects as parts of a single multipart object upload 5. Delete the previous parsed objects once the multipart object upload is complete (a delete marker should be added to the top of the version stack, but the previously stored version should still be available if you specify its handle/version id). Edit: changed formatting