3 ms·

Sounds like something to use Kinesis[1] for. Read the kinesis stream, process the updates and write/rewrite the S3 file. 1 - https://aws.amazon.com/kinesis/ ht

• jxf 12y ago
  Repeatedly updating S3 files on a stream that produced a lot of updates (e.g. 1-second-resolution server logs) would get very expensive, very quickly! Something like that might work if you only periodically rewrote the S3 file (e.g. once an hour). But it's still much less efficient than just allowing delta updates.
  • toomuchtodo 12y ago
    You'd want to write your logs to S3, and then process and cache in-memory, either in something like Redis or in-memory in your analytics app on an EC2 instance.