4 ms·

I followed the exporting to S3 approach. Unfortunately it takes a while until those files become available, even though the AWS API says they are. We had many

I followed the exporting to S3 approach. Unfortunately it takes a while until those files become available, even though the AWS API says they are. We had many issues with this approach in the past. Better host the response yourself....

robjan 5y ago
As of just over a year ago they now provide strong read after write consistency which should solve this problem.
- biermic 5y ago
  Just ran into the problem again last week. Even though we verify the files availability using the boto3 client...
loevborg 5y ago
Once the initial S3 PUT request succeeds, the object is guaranteed to be available immediately AFAIK.
- cle 5y ago
  As long as you haven’t tried to GET the object before the PUT succeeded. Otherwise there can be some eventual consistency for the negative cache to be overwritten. This can be quite annoying if you have e.g. a separate process polling for an S3 object. The object can appear in a HEAD and then return a 404 on the subsequent GET.
  - loevborg 5y ago
    You're right, I encountered this issue as well a few years ago. However, this has changed recently. S3 now offers strong consistency https://aws.amazon.com/s3/consistency/ https://aws.amazon.com/s3/consistency/
  - throwaway858 5y ago
    As mentioned in the sibling comment, this has been fixed by AWS a few months ago and now all S3 operations are always strongly consistent (not eventually consistent)
    - biermic 5y ago
      Just happened again last week. What's even worse the location from where we tried to access the file also had impact. Like we were able to reproduce the issue from one location, but not from another.
      - loevborg 5y ago
        That's interesting - have you tried contacting support? I don't think this is supposed to happen. You might also have better luck using the region-specific S3 endpoint rather than the global domain name.
        biermic 5y ago
        That's a fair point. I'll switch over to region specific! Thanks for putting me in that direction. And no - haven't contacted support so far. Will assign someone from my team to get in touch with them.
  - biermic 5y ago
    It sometimes takes minutes for the file to become available. Meanwhile we get a 401 unauthorized. Even though we have something in place like that: def is_file_uploaded(file_name, request_id): for i in range(S3_UPLOAD_ATTEMPTS_COUNT): try: s3.Object(BUCKET, file_name).load() return True except botocore.exceptions.ClientError as e: logger.warning('File still not uploaded to S3. Check - {}/{} (request_id - {})'.format( i+1, S3_UPLOAD_ATTEMPTS_COUNT, request_id )) return False
- biermic 5y ago
  Nope it doesn't :(
adreamingsoul 5y ago
Ah, good old eventual consistency.

robjan 5y ago

As of just over a year ago they now provide strong read after write consistency which should solve this problem.

biermic 5y ago
Just ran into the problem again last week. Even though we verify the files availability using the boto3 client...

loevborg 5y ago

Once the initial S3 PUT request succeeds, the object is guaranteed to be available immediately AFAIK.

cle 5y ago
As long as you haven’t tried to GET the object before the PUT succeeded. Otherwise there can be some eventual consistency for the negative cache to be overwritten. This can be quite annoying if you have e.g. a separate process polling for an S3 object. The object can appear in a HEAD and then return a 404 on the subsequent GET.
- loevborg 5y ago
  You're right, I encountered this issue as well a few years ago. However, this has changed recently. S3 now offers strong consistency https://aws.amazon.com/s3/consistency/ https://aws.amazon.com/s3/consistency/
- throwaway858 5y ago
  As mentioned in the sibling comment, this has been fixed by AWS a few months ago and now all S3 operations are always strongly consistent (not eventually consistent)
  - biermic 5y ago
    Just happened again last week. What's even worse the location from where we tried to access the file also had impact. Like we were able to reproduce the issue from one location, but not from another.
    - loevborg 5y ago
      That's interesting - have you tried contacting support? I don't think this is supposed to happen. You might also have better luck using the region-specific S3 endpoint rather than the global domain name.
      - biermic 5y ago
        That's a fair point. I'll switch over to region specific! Thanks for putting me in that direction. And no - haven't contacted support so far. Will assign someone from my team to get in touch with them.
- biermic 5y ago
  It sometimes takes minutes for the file to become available. Meanwhile we get a 401 unauthorized. Even though we have something in place like that: def is_file_uploaded(file_name, request_id): for i in range(S3_UPLOAD_ATTEMPTS_COUNT): try: s3.Object(BUCKET, file_name).load() return True except botocore.exceptions.ClientError as e: logger.warning('File still not uploaded to S3. Check - {}/{} (request_id - {})'.format( i+1, S3_UPLOAD_ATTEMPTS_COUNT, request_id )) return False
biermic 5y ago
Nope it doesn't :(

adreamingsoul 5y ago

Ah, good old eventual consistency.