4 ms·

Once the initial S3 PUT request succeeds, the object is guaranteed to be available immediately AFAIK.

cle 5y ago
As long as you haven’t tried to GET the object before the PUT succeeded. Otherwise there can be some eventual consistency for the negative cache to be overwritten. This can be quite annoying if you have e.g. a separate process polling for an S3 object. The object can appear in a HEAD and then return a 404 on the subsequent GET.
- loevborg 5y ago
  You're right, I encountered this issue as well a few years ago. However, this has changed recently. S3 now offers strong consistency https://aws.amazon.com/s3/consistency/ https://aws.amazon.com/s3/consistency/
- throwaway858 5y ago
  As mentioned in the sibling comment, this has been fixed by AWS a few months ago and now all S3 operations are always strongly consistent (not eventually consistent)
  - biermic 5y ago
    Just happened again last week. What's even worse the location from where we tried to access the file also had impact. Like we were able to reproduce the issue from one location, but not from another.
    - loevborg 5y ago
      That's interesting - have you tried contacting support? I don't think this is supposed to happen. You might also have better luck using the region-specific S3 endpoint rather than the global domain name.
      - biermic 5y ago
        That's a fair point. I'll switch over to region specific! Thanks for putting me in that direction. And no - haven't contacted support so far. Will assign someone from my team to get in touch with them.
- biermic 5y ago
  It sometimes takes minutes for the file to become available. Meanwhile we get a 401 unauthorized. Even though we have something in place like that: def is_file_uploaded(file_name, request_id): for i in range(S3_UPLOAD_ATTEMPTS_COUNT): try: s3.Object(BUCKET, file_name).load() return True except botocore.exceptions.ClientError as e: logger.warning('File still not uploaded to S3. Check - {}/{} (request_id - {})'.format( i+1, S3_UPLOAD_ATTEMPTS_COUNT, request_id )) return False
biermic 5y ago
Nope it doesn't :(

cle 5y ago

As long as you haven’t tried to GET the object before the PUT succeeded. Otherwise there can be some eventual consistency for the negative cache to be overwritten. This can be quite annoying if you have e.g. a separate process polling for an S3 object. The object can appear in a HEAD and then return a 404 on the subsequent GET.

loevborg 5y ago
You're right, I encountered this issue as well a few years ago. However, this has changed recently. S3 now offers strong consistency https://aws.amazon.com/s3/consistency/ https://aws.amazon.com/s3/consistency/
throwaway858 5y ago
As mentioned in the sibling comment, this has been fixed by AWS a few months ago and now all S3 operations are always strongly consistent (not eventually consistent)
- biermic 5y ago
  Just happened again last week. What's even worse the location from where we tried to access the file also had impact. Like we were able to reproduce the issue from one location, but not from another.
  - loevborg 5y ago
    That's interesting - have you tried contacting support? I don't think this is supposed to happen. You might also have better luck using the region-specific S3 endpoint rather than the global domain name.
    - biermic 5y ago
      That's a fair point. I'll switch over to region specific! Thanks for putting me in that direction. And no - haven't contacted support so far. Will assign someone from my team to get in touch with them.
biermic 5y ago
It sometimes takes minutes for the file to become available. Meanwhile we get a 401 unauthorized. Even though we have something in place like that: def is_file_uploaded(file_name, request_id): for i in range(S3_UPLOAD_ATTEMPTS_COUNT): try: s3.Object(BUCKET, file_name).load() return True except botocore.exceptions.ClientError as e: logger.warning('File still not uploaded to S3. Check - {}/{} (request_id - {})'.format( i+1, S3_UPLOAD_ATTEMPTS_COUNT, request_id )) return False

biermic 5y ago

Nope it doesn't :(