4 ms·
The versioning is cool in and of itself, but auto-archiving to Glacier after an expiration date is a fantastic feature. The value-add on that is going to be rea
by jxf 12y ago
The versioning is cool in and of itself, but auto-archiving to Glacier after an expiration date is a fantastic feature. The value-add on that is going to be really great for a lot of teams I work with.
What I'd really love to see on S3, though, is the ability to make partial or append-only updates without needing to re-upload the entire bitstream. As of the last time I checked this wasn't directly possible at the API level (it seems like most people store deltas and "reconstruct" the file programmatically when they need to do this).
This would enable an entire class of use cases that's otherwise not easy to do on S3, but which I think would be useful (streaming logs, sensor data, etc.).
- pestaa 12y agoThat'd seriously put logging startups in a tough position, but I guess processing the logs is still the main differentiator which you must do by hand on S3.
- leef 12y agoSounds like something to use Kinesis[1] for. Read the kinesis stream, process the updates and write/rewrite the S3 file. 1 - https://aws.amazon.com/kinesis/ https://aws.amazon.com/kinesis/
- jxf 12y agoRepeatedly updating S3 files on a stream that produced a lot of updates (e.g. 1-second-resolution server logs) would get very expensive, very quickly! Something like that might work if you only periodically rewrote the S3 file (e.g. once an hour). But it's still much less efficient than just allowing delta updates.
- toomuchtodo 12y agoYou'd want to write your logs to S3, and then process and cache in-memory, either in something like Redis or in-memory in your analytics app on an EC2 instance.