4 ms·

> The correct solution, according to our support contact, is that we wait a second after uploading to S3 I'm shocked that you got this answer. This is definite

> The correct solution, according to our support contact, is that we wait a second after uploading to S3 I'm shocked that you got this answer. This is definitely not how you are supposed to operate. If you need to ensure the sequantiality of a write followed by a read on S3, the idiomatic way is to enable versioning on your bucket, issue a write, and provide the version ID to whoever need to read after that write. Not only will that transparently ensure that you will not read deprecated data, but it will even ensure that you actually read the result of that particular write, and not any consecutive write that could have happened in between. This pattern is very easy to implement for sequential operations in a single function, like: version = s3.put_object(...) data = s3.get_object(version_id=version, ...) I agree that when you deal with various processes it can become messy. In practice it never bothered me too much though. I prefer having explicit synchronization through versions, rather than having a blocking write waiting for all caches to be synchronized. Also, this should only be necessary if you write to an already existing object. New keys will always trigger a cache invalidation from my understanding.

fxtentacle 6y ago
Depending on your write throughput, versioning can be quite expensive. Also, we were already at a throughput where the s3 metadata nodes would sometimes go offline, so I'm not sure putting more work on them would have improved our overall situation.
- Galanwe 6y ago
  Interesting, do you have more information on how you did monitors these S3 metadata nodes? Never heard of that. Why would an unversioned put be less costly than a versioned put? To me it should be pretty much the same. I would almost suspect unversioned puts are hacked around versioned ones. Out of curiosity what kind of workload did you perform? I have been moving terabytes of files (some very small 1kB files, some huge 150GB files) and never noticed a single blip of change in behavior from S3 (and certainly not anything going "off")
  - fxtentacle 6y ago
    > Interesting, do you have more information on how you did monitors these S3 metadata nodes? Never heard of that. Pretty simple, we started a bulk import, and then HTTPS requests to amazonaws.com started timing out... Later, support told us to spread out our requests to alleviate the load spike. What we did was process roughly 50mio 1MB JPEG images from the full-res S3 bucket into an derived-data S3 bucket.

fxtentacle 6y ago

Depending on your write throughput, versioning can be quite expensive. Also, we were already at a throughput where the s3 metadata nodes would sometimes go offline, so I'm not sure putting more work on them would have improved our overall situation.

Galanwe 6y ago
Interesting, do you have more information on how you did monitors these S3 metadata nodes? Never heard of that. Why would an unversioned put be less costly than a versioned put? To me it should be pretty much the same. I would almost suspect unversioned puts are hacked around versioned ones. Out of curiosity what kind of workload did you perform? I have been moving terabytes of files (some very small 1kB files, some huge 150GB files) and never noticed a single blip of change in behavior from S3 (and certainly not anything going "off")
- fxtentacle 6y ago
  > Interesting, do you have more information on how you did monitors these S3 metadata nodes? Never heard of that. Pretty simple, we started a bulk import, and then HTTPS requests to amazonaws.com started timing out... Later, support told us to spread out our requests to alleviate the load spike. What we did was process roughly 50mio 1MB JPEG images from the full-res S3 bucket into an derived-data S3 bucket.