3 ms·
I’m still trying to understand if the lack of the PutIfAbsent API in S3 (in other words, the ability to put an item only if no other item is present) is a conse
by Lucasoato 3y ago
I’m still trying to understand if the lack of the PutIfAbsent API in S3 (in other words, the ability to put an item only if no other item is present) is a consequence of the strong consistency they offer in terms of Read after Write. Is this trade-off a corollary of the Cap theorem?
That PutIfAbsent Api would solve so many problems for metadata formats used in the BigData field like Delta or Iceberg!
- bostik 3y agoPossibly a side effect from S3's eventual consistency model. Until couple of years ago there was no guarantee that an object you had written to S3 would be visible to another process looking for it. About two years ago AWS made a big announcement that they could finally guarantee "read your own writes" consistency model. But ONLY within a single account. If you know that you are not racing against eventual consistency, you can use HeadObject API call to check whether a given key exists in the bucket or not.
- gregw2 3y ago"Until couple of years ago there was no guarantee that an object you had written to S3 would be visible to another process looking for it." Yeah, over the last 5 years but mostly before that change, I saw some strange intermittent problems with Hive where certain files would be registered in the Hive metadata by one process but not visible when the data actually went to get queried by other processes which led to jobs we were responsible for bombing out with missing file errors. What was truly strange was when this situation would last for hours.