4 ms·

Possibly a side effect from S3's eventual consistency model. Until couple of years ago there was no guarantee that an object you had written to S3 would be visi

by bostik 3y ago

Possibly a side effect from S3's eventual consistency model. Until couple of years ago there was no guarantee that an object you had written to S3 would be visible to another process looking for it. About two years ago AWS made a big announcement that they could finally guarantee "read your own writes" consistency model. But ONLY within a single account. If you know that you are not racing against eventual consistency, you can use HeadObject API call to check whether a given key exists in the bucket or not.

gregw2 3y ago
"Until couple of years ago there was no guarantee that an object you had written to S3 would be visible to another process looking for it." Yeah, over the last 5 years but mostly before that change, I saw some strange intermittent problems with Hive where certain files would be registered in the Hive metadata by one process but not visible when the data actually went to get queried by other processes which led to jobs we were responsible for bombing out with missing file errors. What was truly strange was when this situation would last for hours.