3 ms·

> We since have moved to a multi-way handshake-like upload process that uses signed URLs vendored to the clients upon request so they can upload the files direc

by bambax 9y ago

> We since have moved to a multi-way handshake-like upload process that uses signed URLs vendored to the clients upon request so they can upload the files directly to S3. How does this work in practice / where can one learn more about this?

CryoLogic 9y ago
Client makes request to server passing back auth token, server verifies auth token and uses the S3 library to generate a unique 1 time use URL for upload directly to the client. Client makes a put request to the s3 url. After it's finished s3 revokes the URL. Multipart signed upload is much harder and requires signing every chunk. Just google s3 signed upload there are a few tutorials from Amazon.
sergiotapia 9y ago
It's actually really simple and quick to do. You offload a ton of processing power to S3. It works like this: User tells the backend, “I want to upload picture.jpeg!” Backend tells the user, “Alright you have my permission but ONLY for that filename with that extension. Here’s a token, enjoy.” User uses that signed token and pushes the file to your S3 bucket. Here's how you do it in Phoenix. https://sergiotapia.me/phoenix-framework-uploading-to-amazon-s3-e70657bd2013 https://sergiotapia.me/phoenix-framework-uploading-to-amazon...
theIV 9y ago
http://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlUploadObject.html http://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlU... Not 100% sure what they mean by _vendored_ here, but I'm guessing they make a request to one of their backends to generate the URL and return it to the client for use.
rawnlq 9y ago
I want to make sure that I understand the security aspect of this. You can argue that the user can upload anything using the original api anyway. But in the original case you can do server-side validation before the upload is proxied. I am thinking stuff that are domain specific like only allowing videos that are 6 seconds long or something. You can move the validation to the client but the client can be easily modified. An actual user might not do this but someone trying steal your storage space (for serving malware or something) might? These signed urls also seem to expire based on time so you can potentially save the url and upload again later if you allow generous expiration. (again, not really something I see being a huge problem) But I guess these aren't really serious issues compared to the cost savings. Am I missing other ways this can be exploited? I am looking into the GCS version, not S3, if that matters: https://cloud.google.com/storage/docs/access-control/signed-urls https://cloud.google.com/storage/docs/access-control/signed-...
- ryanworl 9y ago
  You would use two buckets in this case. Input bucket gets consumed by worker processes to do the transcoding (and validation) and then they upload into the output bucket. The output bucket is what you serve to clients (hopefully with a CDN in front).
  - rawnlq 9y ago
    Thanks! This is a great solution but none of the tutorials/blogs I read on pre-signed uploads mentions it. Do you have links (or just keywords) to learn more? Will I need to add something like Cloud Pub/Sub to my stack? https://cloud.google.com/solutions/using-cloud-pub-sub-long-running-tasks https://cloud.google.com/solutions/using-cloud-pub-sub-long-... This is more complicated than I imagined so I am not sure the cost saving will still work out (factoring in development time and extra code maintenance cost).
    - ryanworl 9y ago
      No, I don’t, sorry. What I can promise you is that you’ll thank yourself for implementing it! There is hardly any additional complexity here because you’d probably be uploading the derived content somewhere anyway. Now you’re just putting in a different place than the source. You can use whatever queue you’re comfortable with so long as you can pipe the upload events from the bucket into it. The pattern I’m outlining is just a physical separation of buckets to make access control much harder to screw up.
    - TheReveller 9y ago
      I can comment on using pub/sub - it's an immensely useful abstraction for these kinds of tasks and something that is quite difficult to implement yourself with the same level of guarantees that using the cloud service will provide. Any time you need to pass information or trigger events asynchronously messaging is the first choice IMO.
antoncohen 9y ago
Heroku has good docs on this, for multiple languages: https://devcenter.heroku.com/articles/s3#file-uploads https://devcenter.heroku.com/articles/s3#file-uploads One thing to keep in mind, users should be able to upload (to the specific signed URL), they should not be able to download from that location. Don't make the files users can upload publicly downloadable, otherwise you can be used to host malware. After the video/image is uploaded, you need to download and process it[1], then upload it to an S3 bucket that allows download (e.g, via CDN). [1] Use caution when processing user content. It is best to process media in a sandbox that can protect you against exploits in the media processing libraries.

by bambax 9y ago

CryoLogic 9y ago

Client makes request to server passing back auth token, server verifies auth token and uses the S3 library to generate a unique 1 time use URL for upload directly to the client. Client makes a put request to the s3 url. After it's finished s3 revokes the URL. Multipart signed upload is much harder and requires signing every chunk. Just google s3 signed upload there are a few tutorials from Amazon.

sergiotapia 9y ago

It's actually really simple and quick to do. You offload a ton of processing power to S3. It works like this: User tells the backend, “I want to upload picture.jpeg!” Backend tells the user, “Alright you have my permission but ONLY for that filename with that extension. Here’s a token, enjoy.” User uses that signed token and pushes the file to your S3 bucket. Here's how you do it in Phoenix. https://sergiotapia.me/phoenix-framework-uploading-to-amazon-s3-e70657bd2013 https://sergiotapia.me/phoenix-framework-uploading-to-amazon...

theIV 9y ago

http://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlUploadObject.html http://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlU... Not 100% sure what they mean by _vendored_ here, but I'm guessing they make a request to one of their backends to generate the URL and return it to the client for use.

rawnlq 9y ago

I want to make sure that I understand the security aspect of this. You can argue that the user can upload anything using the original api anyway. But in the original case you can do server-side validation before the upload is proxied. I am thinking stuff that are domain specific like only allowing videos that are 6 seconds long or something. You can move the validation to the client but the client can be easily modified. An actual user might not do this but someone trying steal your storage space (for serving malware or something) might? These signed urls also seem to expire based on time so you can potentially save the url and upload again later if you allow generous expiration. (again, not really something I see being a huge problem) But I guess these aren't really serious issues compared to the cost savings. Am I missing other ways this can be exploited? I am looking into the GCS version, not S3, if that matters: https://cloud.google.com/storage/docs/access-control/signed-urls https://cloud.google.com/storage/docs/access-control/signed-...

ryanworl 9y ago
You would use two buckets in this case. Input bucket gets consumed by worker processes to do the transcoding (and validation) and then they upload into the output bucket. The output bucket is what you serve to clients (hopefully with a CDN in front).
- rawnlq 9y ago
  Thanks! This is a great solution but none of the tutorials/blogs I read on pre-signed uploads mentions it. Do you have links (or just keywords) to learn more? Will I need to add something like Cloud Pub/Sub to my stack? https://cloud.google.com/solutions/using-cloud-pub-sub-long-running-tasks https://cloud.google.com/solutions/using-cloud-pub-sub-long-... This is more complicated than I imagined so I am not sure the cost saving will still work out (factoring in development time and extra code maintenance cost).
  - ryanworl 9y ago
    No, I don’t, sorry. What I can promise you is that you’ll thank yourself for implementing it! There is hardly any additional complexity here because you’d probably be uploading the derived content somewhere anyway. Now you’re just putting in a different place than the source. You can use whatever queue you’re comfortable with so long as you can pipe the upload events from the bucket into it. The pattern I’m outlining is just a physical separation of buckets to make access control much harder to screw up.
  - TheReveller 9y ago
    I can comment on using pub/sub - it's an immensely useful abstraction for these kinds of tasks and something that is quite difficult to implement yourself with the same level of guarantees that using the cloud service will provide. Any time you need to pass information or trigger events asynchronously messaging is the first choice IMO.

antoncohen 9y ago

Heroku has good docs on this, for multiple languages: https://devcenter.heroku.com/articles/s3#file-uploads https://devcenter.heroku.com/articles/s3#file-uploads One thing to keep in mind, users should be able to upload (to the specific signed URL), they should not be able to download from that location. Don't make the files users can upload publicly downloadable, otherwise you can be used to host malware. After the video/image is uploaded, you need to download and process it[1], then upload it to an S3 bucket that allows download (e.g, via CDN). [1] Use caution when processing user content. It is best to process media in a sandbox that can protect you against exploits in the media processing libraries.