4 ms·

May I ask you why? I use and have used both PUT presigned urls and POST signed policies for uploading file to S3 without too many problems, but your remark worr

May I ask you why? I use and have used both PUT presigned urls and POST signed policies for uploading file to S3 without too many problems, but your remark worries me a bit. Am I missing something?

gw98 4y ago
We get our customers to upload to S3 directly so we don't have to handle their ingress traffic through our infra. Problems involve mostly when the client cocks up as it's near impossible to distinguish between broken files and good ones. This leads to the assumption that the clients have uploaded a file and they haven't.
- alserio 4y ago
  Do you mean that you cannot synchronously validate the file and return an error to the client? For async validations a lambda on an s3 trigger works well enough. For sync validation I've found you are limited in what you can encode in an upload policy
  - gw98 4y ago
    Can't do that. The clients own the S3 buckets not us. The whole idea we had was stupid.
    - alserio 4y ago
      You still can use POST policies, they don't need to be defined on the bucket but only signed. But you are limited on what you can check. Other than that, yeah, the approach might not have been adequate.
- orf 4y ago
  If you know the file hash beforehand, you can encode that into the signed URL. But if you’re saying clients will upload the wrong file, then that’s not really AWS’s fault is it? If you’re saying the file itself is corrupted, I’m not sure how? The upload should fail if the content length is incorrect, and s3 doesn’t make the contents visible unless the upload completes.
  - gw98 4y ago
    It's the eventual consistency nature of the problem. You have to check and you can't guarantee. Don't expect the clients to work properly either. Edge notoriously started uploading zero byte files when already open in MS office a few months back. Also network connections time out, users close browser tabs etc etc.

May I ask you why? I use and have used both PUT presigned urls and POST signed policies for uploading file to S3 without too many problems, but your remark worries me a bit. Am I missing something?

gw98 4y ago

We get our customers to upload to S3 directly so we don't have to handle their ingress traffic through our infra. Problems involve mostly when the client cocks up as it's near impossible to distinguish between broken files and good ones. This leads to the assumption that the clients have uploaded a file and they haven't.

alserio 4y ago
Do you mean that you cannot synchronously validate the file and return an error to the client? For async validations a lambda on an s3 trigger works well enough. For sync validation I've found you are limited in what you can encode in an upload policy
- gw98 4y ago
  Can't do that. The clients own the S3 buckets not us. The whole idea we had was stupid.
  - alserio 4y ago
    You still can use POST policies, they don't need to be defined on the bucket but only signed. But you are limited on what you can check. Other than that, yeah, the approach might not have been adequate.
orf 4y ago
If you know the file hash beforehand, you can encode that into the signed URL. But if you’re saying clients will upload the wrong file, then that’s not really AWS’s fault is it? If you’re saying the file itself is corrupted, I’m not sure how? The upload should fail if the content length is incorrect, and s3 doesn’t make the contents visible unless the upload completes.
- gw98 4y ago
  It's the eventual consistency nature of the problem. You have to check and you can't guarantee. Don't expect the clients to work properly either. Edge notoriously started uploading zero byte files when already open in MS office a few months back. Also network connections time out, users close browser tabs etc etc.