5 ms·
How 500px serves up 500TB of high res photos
- ranrub 11y agoPopular photo sharing company opens the kimono on their photo processing stack.
- dataminer 11y agoThumbs up for VIPS, used it 4-5 years ago, for building a system which processed high resolution images. Its fast, lean and an amazing piece of software.
- Melraidin 11y agoVIPS has been great to work with. The docs are generally very good and the performance has been great. The author (https://github.com/jcupitt/ https://github.com/jcupitt/) has always been responsive to issues and questions and seems to be constantly working on the project. My only, relatively minor quibble, is dev work happening on master, but when it's largely a one-man project who am I to judge. So far we haven't come across any problems that weren't either self-inflicted or already fixed in master.
- Renat 11y agoPerformace is the reason why we switched from ImageMagick
- deleted 11y ago[deleted]
- fotios 11y agoTotally agree, I've used it for some projects as well. The only thing I wasn't able to grok (which still kills me to this day), was that I couldn't get it to handle EPS files correctly. If anybody has done that with VIPS/IM, I would love to hear about it.
- Melraidin 11y agoVIPS should be able to open EPS files via libmagick but at least in my test I found that I ended up with just a black box. Maybe related to the below bug? https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=170649 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=170649
- jeffreyrogers 11y agoI'm replying here to let Melraidin (who posted a sibling comment) know that their account is dead, for reasons I can't understand since their comments seem helpful.
- quasse 11y agoAre they somehow intercepting middle click on their site so that links do not open in a new tab? Control-click works fine for me, but middle click would just replace the article, which was inconvenient. (latest Chrome on Ubuntu)
- aroch 11y agoMiddle-click seems to be working fine here (FF41, OSX)
- leftnode 11y agoI'm getting the same thing and it's incredibly frustrating on an otherwise interesting article. I had to Right Click > "Open In New Tab" for links I wanted to read later.
- distantsounds 11y agoSame here, Chrome on W7.
- Retr0spectrum 11y agoWorks for me on Firefox 38 on Arch Linux.
- nols 11y agoSame for me (Chrome on Mint)
- callumjones 11y agoInteresting that they upload to S3 first, only to pull it off soon after and process it. I imagine this reduces complexity and lets them increase their throughput of uploads but I wonder if 500px toyed around with the idea of first uploading it to local (temp) storage, having the workers process it and then finally persist it. But I guess the cons are a) some point it needs to go to S3 in original form b) increases data loss
- ranrub 11y agoIt also means that you need to scale your upload servers (IO & long requests), this is more difficult than trusting S3 to scale for you. However having your S3 open to uploads has it's downsides. We at Cloudinary do uploads to our own (autoscaled) instances and only then persist.
- yeukhon 11y agoTo me this is a very common practice. At least in my experience a lot of AWS processing are done with S3 throughout the pipeline, whether you run EMR or run simple workflow. You can also create API/temp credential per user in your app allows the user to upload things to "storage" (but in this case S3) and process from there.
- ranrub 11y agoAnd with Lambda based on S3 events, you can run stuff automatically on that as well. However, if you want your uploaders to get feedback on their uploads immediately and/or synchronously, you need to upload to app servers.
- Melraidin 11y agoI'd like to be able to have users upload directly to S3 (obviously less infrastructure and code to maintain) but without being able to provide immediate feedback on the upload I've found it to be preferable to have our own application servers in the upload path. This lets us immediately detect unsupported formats and act on the completion of the upload (whether successful or failure) without delay.
- happytrails 11y agoAll the services! An Engineer's paradise.
- cconcepts 11y agoI was getting excited about using 500px for the specific reason that it seems to load much faster than flickr. Then I loaded it up in a public space to get some inspiration by browsing other user's work - interspersed were various images of women in various states of undress. I quickly exited the site and never went back for fear of someone glancing over my shoulder and assuming I'm doing something other than getting artistic inspiration. Not arguing that the images aren't art but I personally don't want them popping up on my screen when I'm just trying to get inspiration for landscape photography.
- dagw 11y ago500px at least used to have a feature to autofilter out nude photos.