5 ms·

Interesting that they upload to S3 first, only to pull it off soon after and process it. I imagine this reduces complexity and lets them increase their through

Interesting that they upload to S3 first, only to pull it off soon after and process it. I imagine this reduces complexity and lets them increase their throughput of uploads but I wonder if 500px toyed around with the idea of first uploading it to local (temp) storage, having the workers process it and then finally persist it. But I guess the cons are a) some point it needs to go to S3 in original form b) increases data loss

ranrub 11y ago
It also means that you need to scale your upload servers (IO & long requests), this is more difficult than trusting S3 to scale for you. However having your S3 open to uploads has it's downsides. We at Cloudinary do uploads to our own (autoscaled) instances and only then persist.
yeukhon 11y ago
To me this is a very common practice. At least in my experience a lot of AWS processing are done with S3 throughout the pipeline, whether you run EMR or run simple workflow. You can also create API/temp credential per user in your app allows the user to upload things to "storage" (but in this case S3) and process from there.
- ranrub 11y ago
  And with Lambda based on S3 events, you can run stuff automatically on that as well. However, if you want your uploaders to get feedback on their uploads immediately and/or synchronously, you need to upload to app servers.
  - Melraidin 11y ago
    I'd like to be able to have users upload directly to S3 (obviously less infrastructure and code to maintain) but without being able to provide immediate feedback on the upload I've found it to be preferable to have our own application servers in the upload path. This lets us immediately detect unsupported formats and act on the completion of the upload (whether successful or failure) without delay.
e12e 11y ago
My initial thought was: wow, this looks like a great validation of the idea behind Joyent Manta[1] storage+process. I wonder if it would've been a good fit for 500px, and how it would compare in terms of price/performance? Granted, they now have a working system on s3, so maybe for the next up-and-comming competitor? ;-) Great write-up, btw. And thanks for the heads-up about vips/nip2 -- I wasn't aware of those. [1] https://www.joyent.com/object-storage https://www.joyent.com/object-storage
Renat 11y ago
I'm the CTO of 500px. Decoupling uploaders and converters simplifies the overall design and makes it more robust. Converters are a part of the legacy design that we are planning to retire. In the past, we would generate multiple sizes and crops for every image so it can be efficiently displaced on the site and/or mobile apps. Today, we rely on the new Resizer service that can resize and crop images on the fly. It's an interesting piece of technology that we will be writing about on our blog: developers.500px.com
- bliti 11y ago
  I will keep posted to your blog. Thanks for sharing the link. Always looking to learn about memory heavy operations. Currently am the lead dev on software that revolves around user generated content (images and audio). Its an ever ending road of (fun) challenges.
- discardorama 11y ago
  I am surprised that it is faster to resize the image, than to pass the (resized) bits right through. Is the outbound network a bottleneck?
  - kalleboo 11y ago
    It could be that there are a lot of images/versions that simply never get seen, or seen rarely enough that it's inefficient to store them. And they're being cached by the CDN anyway, so popular images will still only be generated once per edge.
    - Renat 11y ago
      bingo! Extra storage for all the resized images that never get seen, lack of flexibility to introduce new sizes, plus requirements for watermarking and attribution on images are the main reasons why we are moving away from pre-conversion to dynamic image resizing
  - Melraidin 11y ago
    Generating many conversions for all photos incurs both storage and computational overhead that may never pay off. By only generating the required images on the fly for photos we avoid this overhead while allowing more flexibility. If watermarks or attribution change in the future it's easier to have the resize service in place to handle this as images expire from the CDN than to do a batch job to pre-process the updated conversion requirements for all existing photos. Also note that photos are stored indefinitely, so any additional permanent storage (new conversions) would add additional storage costs, miniscule as they might be, forever.

ranrub 11y ago

It also means that you need to scale your upload servers (IO & long requests), this is more difficult than trusting S3 to scale for you. However having your S3 open to uploads has it's downsides. We at Cloudinary do uploads to our own (autoscaled) instances and only then persist.

yeukhon 11y ago

To me this is a very common practice. At least in my experience a lot of AWS processing are done with S3 throughout the pipeline, whether you run EMR or run simple workflow. You can also create API/temp credential per user in your app allows the user to upload things to "storage" (but in this case S3) and process from there.

ranrub 11y ago
And with Lambda based on S3 events, you can run stuff automatically on that as well. However, if you want your uploaders to get feedback on their uploads immediately and/or synchronously, you need to upload to app servers.
- Melraidin 11y ago
  I'd like to be able to have users upload directly to S3 (obviously less infrastructure and code to maintain) but without being able to provide immediate feedback on the upload I've found it to be preferable to have our own application servers in the upload path. This lets us immediately detect unsupported formats and act on the completion of the upload (whether successful or failure) without delay.

e12e 11y ago

My initial thought was: wow, this looks like a great validation of the idea behind Joyent Manta[1] storage+process. I wonder if it would've been a good fit for 500px, and how it would compare in terms of price/performance? Granted, they now have a working system on s3, so maybe for the next up-and-comming competitor? ;-) Great write-up, btw. And thanks for the heads-up about vips/nip2 -- I wasn't aware of those. [1] https://www.joyent.com/object-storage https://www.joyent.com/object-storage

Renat 11y ago

I'm the CTO of 500px. Decoupling uploaders and converters simplifies the overall design and makes it more robust. Converters are a part of the legacy design that we are planning to retire. In the past, we would generate multiple sizes and crops for every image so it can be efficiently displaced on the site and/or mobile apps. Today, we rely on the new Resizer service that can resize and crop images on the fly. It's an interesting piece of technology that we will be writing about on our blog: developers.500px.com

bliti 11y ago
I will keep posted to your blog. Thanks for sharing the link. Always looking to learn about memory heavy operations. Currently am the lead dev on software that revolves around user generated content (images and audio). Its an ever ending road of (fun) challenges.
discardorama 11y ago
I am surprised that it is faster to resize the image, than to pass the (resized) bits right through. Is the outbound network a bottleneck?
- kalleboo 11y ago
  It could be that there are a lot of images/versions that simply never get seen, or seen rarely enough that it's inefficient to store them. And they're being cached by the CDN anyway, so popular images will still only be generated once per edge.
  - Renat 11y ago
    bingo! Extra storage for all the resized images that never get seen, lack of flexibility to introduce new sizes, plus requirements for watermarking and attribution on images are the main reasons why we are moving away from pre-conversion to dynamic image resizing
- Melraidin 11y ago
  Generating many conversions for all photos incurs both storage and computational overhead that may never pay off. By only generating the required images on the fly for photos we avoid this overhead while allowing more flexibility. If watermarks or attribution change in the future it's easier to have the resize service in place to handle this as images expire from the CDN than to do a batch job to pre-process the updated conversion requirements for all existing photos. Also note that photos are stored indefinitely, so any additional permanent storage (new conversions) would add additional storage costs, miniscule as they might be, forever.