5 ms·

Ha, I thought that was a joke. Apparently not: https://blog.imgix.com/2015/05/08/racking-mac-pros-hardware-design-for-web-scale.html https://blog.imgix.com/201

by rb2k_ 10y ago

Ha, I thought that was a joke. Apparently not: https://blog.imgix.com/2015/05/08/racking-mac-pros-hardware-design-for-web-scale.html https://blog.imgix.com/2015/05/08/racking-mac-pros-hardware-...

brianwawok 10y ago
Can you really not run image rendering code on Linux servers? Vs you know, spending thousands of dollars racking OSX servers? Because that would make "increasing capacity" about 3 clicks in AWS.... Perhaps I am missing the competitive advantage of using OSX to resize images, but it sure seems like the shortcomings are obvious :)
- microcolonel 10y ago
  Well, basically they developed the thing on macbooks with Apple's image manipulation libraries (which are basically just implementations of published, standard algorithms). Then they didn't bother porting it to something they could scale. Instead, they opted to commit to putting genuine Apple hardware on racks, something that even Apple doesn't do. It's an understandable direction if your business is at the "Well, I have a few of these mac pros sitting around, I wonder if I could make a living at this" stage.
  - brianwawok 10y ago
    Sure, great way to POC. Then don't you immediately get funding and go hire some devs to code it up for real? Vs hiring server guys to rack 100s of Macs? Seems like something your VC should suggest to you, not "heres a million bucks, go create rack mount macs.."
    - skuhn 10y ago
      [I work at imgix and have helped lead the team on the production issues we face and gathering the details for this blog post] I've touched on this elsewhere, but here's the crux of the problem: Racking servers, even if they're cylindrical, is actually much easier than building an entire real-time image rendering pipeline from scratch. The chassis that we co-designed and had made works out to a few hundred dollars per machine, since we aren't buying in giant quantity. When you talk about servers that run 24/7 and cost between $5k-25k per box, that's not the dominant expense by any measure. It's also been picked up by some other companies to use, which is always nice to see -- we aren't keeping it as a proprietary solution, anyone can buy it from our vendor. Either way we decided to go, we would still be solving both the image rendering and machine racking / operation problems simultaneously to different degrees. As it sits now, we have about 4-5x the number of infrastructure and imaging engineers as we have datacenter managers / technicians on staff. As it should be, I think. This design decision is actually part of our advantage, in that it allows us to deliver more functionality to our customers at a lower price per unit on the backend. There's always room for improvement, and we continue to iterate on it -- some day the Macs will probably be gone from the datacenter, but only when it's the right move to make.
      - rb2k_ 10y ago
        Since the post mentions performance as one of the main problems, it seems kinda strange to defend the Mac Pro. The trashcan Mac Pro has very outdated Xeon E5620 CPUs and it doesn't seem like there's a lot of upgrades in the pipeline. I can even see Apple stopping the sale of these in favor of iMacs. I think any 'regular' sever with e.g. an Intel Xeon E5-1650 v3 be about twice as fast for basically the same price. I'd assume that image manipulation is mainly a CPU bound issue. > Racking servers, even if they're cylindrical, is actually much easier than building an entire real-time image rendering pipeline from scratch. Scaling a service based on Mac Pros is probably much harder than starting to port pieces of a real-time image rendering pipeline to hardware that is a lot faster and more readily available. To be honest, for the price of the Mac Pros, you can probably run a lot of CPU hours in AWS / ... and skip the whole rack-n-stack. That being said, I'm not the one running an image rendering business, so I'm just armchair quarterbacking :)
        skuhn 10y ago
        I appreciate the feedback. These architecture choices are something that I think about a lot, but for now I'm satisfied that we chose the right path based on our requirements at the time. It is annoying that Apple has let the current Mac Pro rot on the vine to a large extent. This is something that certainly gives me pause and will be factored in to the future direction of our rendering pipeline. There may not be much of a future for image rendering on Macs at imgix, but we have to factor in all the variables when deciding on our future roadmap. The difference between a v2 and v3 CPU definitely isn't double performance, by the way. An E5-2697v2 (the top-of-line Mac Pro CPU) is a 970 on SPECintrate, and an E5-2697v3 is a 1240 on SPECintrate (we track other measurements, but that's the quickest comparison I found). So it's better for sure, but it's also 15W higher draw and has 2 more cores to help grow those numbers. Per-core performance on SPECint is only 60 vs 64. That said, in a perfect world I would love to use v3 or v4 Xeons on our image rendering hosts. It just wouldn't be a night-and-day kind of difference that's worth throwing our existing pipeline out over. Have to be careful not to toss the baby with the bathwater here. The real performance issue for us this time around was adding machines to overcome certain scalability constraints, and optimizing certain code paths that were prohibitively expensive.

by rb2k_ 10y ago

Ha, I thought that was a joke. Apparently not: https://blog.imgix.com/2015/05/08/racking-mac-pros-hardware-design-for-web-scale.html https://blog.imgix.com/2015/05/08/racking-mac-pros-hardware-...

brianwawok 10y ago

Can you really not run image rendering code on Linux servers? Vs you know, spending thousands of dollars racking OSX servers? Because that would make "increasing capacity" about 3 clicks in AWS.... Perhaps I am missing the competitive advantage of using OSX to resize images, but it sure seems like the shortcomings are obvious :)

microcolonel 10y ago
Well, basically they developed the thing on macbooks with Apple's image manipulation libraries (which are basically just implementations of published, standard algorithms). Then they didn't bother porting it to something they could scale. Instead, they opted to commit to putting genuine Apple hardware on racks, something that even Apple doesn't do. It's an understandable direction if your business is at the "Well, I have a few of these mac pros sitting around, I wonder if I could make a living at this" stage.
- brianwawok 10y ago
  Sure, great way to POC. Then don't you immediately get funding and go hire some devs to code it up for real? Vs hiring server guys to rack 100s of Macs? Seems like something your VC should suggest to you, not "heres a million bucks, go create rack mount macs.."
  - skuhn 10y ago
    [I work at imgix and have helped lead the team on the production issues we face and gathering the details for this blog post] I've touched on this elsewhere, but here's the crux of the problem: Racking servers, even if they're cylindrical, is actually much easier than building an entire real-time image rendering pipeline from scratch. The chassis that we co-designed and had made works out to a few hundred dollars per machine, since we aren't buying in giant quantity. When you talk about servers that run 24/7 and cost between $5k-25k per box, that's not the dominant expense by any measure. It's also been picked up by some other companies to use, which is always nice to see -- we aren't keeping it as a proprietary solution, anyone can buy it from our vendor. Either way we decided to go, we would still be solving both the image rendering and machine racking / operation problems simultaneously to different degrees. As it sits now, we have about 4-5x the number of infrastructure and imaging engineers as we have datacenter managers / technicians on staff. As it should be, I think. This design decision is actually part of our advantage, in that it allows us to deliver more functionality to our customers at a lower price per unit on the backend. There's always room for improvement, and we continue to iterate on it -- some day the Macs will probably be gone from the datacenter, but only when it's the right move to make.
    - rb2k_ 10y ago
      Since the post mentions performance as one of the main problems, it seems kinda strange to defend the Mac Pro. The trashcan Mac Pro has very outdated Xeon E5620 CPUs and it doesn't seem like there's a lot of upgrades in the pipeline. I can even see Apple stopping the sale of these in favor of iMacs. I think any 'regular' sever with e.g. an Intel Xeon E5-1650 v3 be about twice as fast for basically the same price. I'd assume that image manipulation is mainly a CPU bound issue. > Racking servers, even if they're cylindrical, is actually much easier than building an entire real-time image rendering pipeline from scratch. Scaling a service based on Mac Pros is probably much harder than starting to port pieces of a real-time image rendering pipeline to hardware that is a lot faster and more readily available. To be honest, for the price of the Mac Pros, you can probably run a lot of CPU hours in AWS / ... and skip the whole rack-n-stack. That being said, I'm not the one running an image rendering business, so I'm just armchair quarterbacking :)
      - skuhn 10y ago
        I appreciate the feedback. These architecture choices are something that I think about a lot, but for now I'm satisfied that we chose the right path based on our requirements at the time. It is annoying that Apple has let the current Mac Pro rot on the vine to a large extent. This is something that certainly gives me pause and will be factored in to the future direction of our rendering pipeline. There may not be much of a future for image rendering on Macs at imgix, but we have to factor in all the variables when deciding on our future roadmap. The difference between a v2 and v3 CPU definitely isn't double performance, by the way. An E5-2697v2 (the top-of-line Mac Pro CPU) is a 970 on SPECintrate, and an E5-2697v3 is a 1240 on SPECintrate (we track other measurements, but that's the quickest comparison I found). So it's better for sure, but it's also 15W higher draw and has 2 more cores to help grow those numbers. Per-core performance on SPECint is only 60 vs 64. That said, in a perfect world I would love to use v3 or v4 Xeons on our image rendering hosts. It just wouldn't be a night-and-day kind of difference that's worth throwing our existing pipeline out over. Have to be careful not to toss the baby with the bathwater here. The real performance issue for us this time around was adding machines to overcome certain scalability constraints, and optimizing certain code paths that were prohibitively expensive.