3 ms·
That S3 is eventually consistent with object updates (HTTP PUT) might also screw up things for a company whose core value is synchronized storage.
by llarsson 6y ago
That S3 is eventually consistent with object updates (HTTP PUT) might also screw up things for a company whose core value is synchronized storage.
- boredpudding 6y agoSince the move to S3 is only 'a few years ago', my guess is this was not the reason for them to move away from S3 since service at Dropbox is pretty much the same.
- RachelF 6y agoI suspect that is behind many problems with Google Drive. Sometimes old versions seems to take precedence over newer ones, for some reason.
- dfsegoat 6y agoI don't mean to sound daft, just clarifying my own understanding, but isn't Dropbox eventually consistent (as a system)?
- llarsson 6y agoOh, sure, but when they think they have written something to S3 and got a successful HTTP response back from the API, perhaps they want to be able to tell clients to go fetch the new data from the bucket. But those clients may not get the new data then, due to eventual consistency.
- lozenge 6y agoS3 is immediately consistent for new objects unless the service received a GET on the object before it was created. It's easy to use this to make an immediately consistent system.
- llarsson 6y agoYes. HTTP POST (new objects) is immediate, HTTP PUT (updates to objects) is eventually consistent. If you want to use this to create new objects all the time, rather than update ones you already have, you now have to keep track of which objects in your bucket are "old" and should no longer be there. But yes, totally doable.
- ozkatz 6y agoS3 ListObjects calls are eventually consistent (i.e. list-after-put). EMRFS [1] and S3Guard [2] mitigate this for data processing use cases. [1] - https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-fs.html https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-f... [2] - https://blog.cloudera.com/introducing-s3guard-s3-consistency-for-apache-hadoop/ https://blog.cloudera.com/introducing-s3guard-s3-consistency...
- ayush--s 6y agoThey might have implemented a metadata store 1 layer before S3 to guarantee read after write consistency since write to a new object is consistent in S3. Only updates are eventually consistent.