9 ms·
How Imgix Built a Stack to Serve 100k Images per Second
- jawngee 11y agoI love imgix to death, really the only web service I use daily that constantly amazes me. I've been using them for a few clients now and it's been a fundamental shift in how we think about media management in the CMS. Rolling out new features with different image size requirements using existing media doesn't require us doing anything other than changing a few parameters in the URL. Before, we'd have to run processing tasks on a fleet of EC2 instances and wait a few days. We compared a few similar services, but none of them really got the flow right like Imgix does. Some require you to setup the parameters beforehand or define everything in their web interface or via a REST api. And quality wise, none of them are near as good as Imgix. I can't sing their praises high enough honestly.
- boundlessdreamz 11y ago> Before, we'd have to run processing tasks on a fleet of EC2 instances and wait a few days. Few days? Why?
- ddorian43 11y ago2 ways to do thumbnails. 1 Precompute(what they did) and 2 Compute on demand(imgix) Of course the best is to do something in the middle.
- boundlessdreamz 11y agoYes. But why should Precompute take a few days?
- Rifu 11y agoPerhaps they have a few million images to go through.
- boundlessdreamz 11y agoAh. I think you are right. Misread the original post. They want to process "existing media". I was thinking of "new media".
- jawngee 11y agoImgix is the middle, they process the parameters once and cache the result.
- ddorian43 11y agoThat is just caching. Middle is: You precompute some fixed presets(like him, lower latency) but also have the ability to generate on demand if a preset doesn't exist(used when adding new presets, no reason to wait *days to have the new preset).
- skuhn 11y agoOnce you've built a system that works fast enough to handle image rendering work on-demand, then adding batch or precomputational features is largely just creating requester logic to ask for the image before the real demand arrives.
- jawngee 11y agoThis particular client has a few hundred thousand images and tens of thousands of animated gifs. It's the animated gifs that take forever because they have to be processed with gifsicle and then converted to mpeg-4. Imgix does all of that. I don't know what kind of voodoo they are doing on the gif crop, but it's lightning fast.
- deleted 11y ago[deleted]
- skuhn 11y agoThis might be more common than you think. I worked at a company a few years ago with ~50 sizes of each image. When we wanted to launch a new page layout (or iOS app, or what have you), it literally required several weeks to render the millions and millions of images to that new size. And of course things didn't work 100%, since some of the originals had mysteriously disappeared or rotted away. Not to mention the on-going storage costs of these 51 copies of each image, the vast majority of which were never requested even a single time. Another approach at a different company was to have thousands of EC2 instances running 24/7 to resize images as users uploaded them and another few thousand to handle the backlog that the first group couldn't get to within their time budget. It got the job done, but it was pretty expensive and time consuming to babysit.
- 811639047892938 11y agoSince you've tried some similar services, are there any that are not quite as smooth, but cheaper? I actually am on Hacker News today because I was googling imgix competitors, and it took me to an older article, then I thought I'd check out the front page to see what's new today, and this was there. I love everything I see about imgix except the 8cents/GB bandwidth, which might be fine for certain sites, but not mine.
- tyho 11y agoHow does the cost of Mac Pro's compare with more traditional servers?
- dubcanada 11y agoThey are more expensive, if you look at mac mini hosting services and compare then to linux you'll see that. But it says that they used them for the graphics processing only. And used "traditional servers" for the other services.
- bluedino 11y agoCheaper than re-writing and re-testing everything to work on Windows/Linux + whitebox servers.
- tacos 11y agoThey're using code that they cannot license elsewhere. They're making OS calls to process the images.
- skuhn 11y agoMy quick and dirty math from a previous HN discussion: https://news.ycombinator.com/item?id=9510610 https://news.ycombinator.com/item?id=9510610 Short version is: not too bad, and we get a lot of utility from OS X frameworks, so the tradeoffs are worth it for us at this stage.
- discardorama 11y ago> An early Facebook photos engineer mentioned he had seen remarkable performance and quality coming out of the Apple Core Graphics stack . What does the Apple stack have, which can't be done on a Linux box with a high-end GPU like a K20 or a Tesla?
- tacos 11y ago[Talking purely CPU] You can resize 100k images a second on a 2013 laptop. [Source: the Google guy who serves your images.] A couple ARM cellphones could meet this entire company's [CPU] image processing needs. Think of what YouTube is doing for videos in the time you read this sentence if you want to put their silly claims in perspective. Anybody remember when Steve Jobs said the 400Mhz PowerPC G4 Altivec was a supercomputer? But the only code that showed up was a Reverb that took a weekend to run and a couple FFT libraries that had accuracy problems? Good times.
- SEJeff 11y agoK20s are old GPUs, get a Tesla K80 :) The apple hardware is either Nvidia or ATI gpus, but the software apis are what are really nice about dealing with some of these things on OS X. I think the Author of the post is confounding new hardware from Nvidia / ATI with new hardware from Apple.
- mrhyperpenguin 11y agoThis is just speculation but probably nothing. Imgix should be doing their image processing with GPGPU technology (CUDA or OpenCL) which should be pretty much available anywhere. Technically Apple has had their Metal API available for a while now but I kind of doubt they're using that for image processing.
- tacos 11y agoThis company repeatedly sets off my B.S. detector with its technology story. Here's a previous article where they didn't have their schtick quite so polished: http://www.wired.com/wiredenterprise/2013/05/imgix-graphics-card-internet/ http://www.wired.com/wiredenterprise/2013/05/imgix-graphics-... Believe it or not: Google already had awesome, high-performance image tech. YouTube re-encoded every video in 2010 as a favor to Apple. Meanwhile these guys are saying Google couldn't make thumbnails? So they had to do a startup using the GPUs in Mac minis? “We just couldn’t crunch those images down to a smaller size,” Zacharias remembers. “It would have taken a significant amount of Google’s entire processing power just to do that.” Preposterous, dishonest claim. I looked at this back then and discovered: Google provides it through App Engine. Adobe provides this through Akamai. http://cloudimage.io/ http://cloudimage.io/ does it through Rackspace CDN. http://cloudinary.com/ http://cloudinary.com/ does it through CloudFront. That said Imgix seems to provide a stable service people enjoy. Perhaps they should sell that instead of these oddball tech claims. Because what they're doing isn't hard or novel. They sure are making it complicated, though.
- 15155 11y ago> They sure are making it complicated, though. Seems like a large degree of it is marketing-related. "Any intelligent fool can make things bigger, more complex, and more violent..."
- dheera 11y agoI wonder if one could slash costs by building a stack on top of Imgix that concatenates several "master" images into a single "grandmaster" image and then just serve them up individually with cropping operations ...
- fasteo 11y agoSomehow disappointed by the title. This is just a list of software they use. IMHO, that's not a "stack" article in my book. >>>> The load balancing and distribution layer is based on custom C code and a LuaJIT framework we created called Levee. Levee is capable of servicing 40K requests per second on a single machine Is "Leeve" open sourced ? It would be nice to see this source code and it would had been great to include some discussion on why they chose to implement this instead of using some other off-the-shelf LB solutions (like HAProxy or Openresty which are also mentioned in the article). This is the kind of things I like seeing in a "stack" article.
- skuhn 11y agoThis is pretty typical in my experience for StackShare posts. For example: http://stackshare.io/500px/how-500px-serves-up-over-500tb-of-high-res-photos http://stackshare.io/500px/how-500px-serves-up-over-500tb-of... imgix is very interested in open sourcing things we create, and there are a few smaller projects at https://github.com/imgix https://github.com/imgix now. Hopefully Levee will join them in the future, once we have reached a certain point in development.
- bpicolo 11y agoMy middle-click is broken on this site and doesn't open things in a new tab. : (
- sul4bh 11y agoFrom the article: "Levee is capable of servicing 40K requests per second on a single machine." How are concurrent connections simulated for testing?
- micheljk 11y ago> "In most cases, your images will traverse the entire stack without ever touching a hard drive." > "Our fetching and caching layers are largely custom built, using MogileFS" Could you specify the use case of mogilefs in your project, is it some sort of "cold storage" for the fetched image data? Can you provide more details - amount of trackers, storage-nodes, storage-size, ... ? Do you also consider using a ceph cluster in future?
- skuhn 11y agoCommodity CDN's will always have some amount of cache misses. For most services that's fine; there isn't too much additional overhead in serving a static Javascript file a few times. If you're doing computational work for each request, you want to try to avoid unnecessary work as much as possible. It's helpful for us to hold onto master images after fetching -- these images can be large and may not be served to us from a particularly fast source. The speed of light does come into play, so anything we can do to shorten distances is beneficial. This cache layer respects the Cache-Control headers of the original objects, so the behavior there is under our user's control. There's a similar need to cache rendered output ("derivative images"), because while most operations are very fast they do still add a little bit of latency. This helps with our rendering capacity, but it's really about improving time-to-serve latency wherever we can. We evaluated a few open source object stores and decided on MogileFS. It largely came down to the simplicity of what Mogile does and how it does it. We figured that we could rip pieces out and adapt it to our needs with a lot less work than more elaborate systems (like ceph). With that said, we didn't actually do an implementation test with ceph, so it may in fact be great for our use case. Not sure off-hand about the number of servers deployed for use with Mogile, but it isn't a gigantic deployment. We aren't storing anything that can't be re-acquired, so we can adjust our storage to hit particular cache hit ratios rather than storing the enormous long-tail forever. The first generation of storage servers are 16 core / 32gb / 6x4T systems -- more like what I would use for computationally related storage like HDFS than a large object store. We could stick with this hardware spec, or we might move to SSDs if latency becomes an issue (I personally doubt it will). We might also move to more dense systems, with more like 20-30 drives. That's all based on our metrics and how things perform in practice.