6 ms·

Three possible reasons I can think of for doing this over using PCs or Linux servers: 1. Using the same operating system as the developers of the software, plu

Three possible reasons I can think of for doing this over using PCs or Linux servers: 1. Using the same operating system as the developers of the software, plus access to Apple's fantastic imaging libraries. 2. The Mac Pro, whilst expensive, is good value for money. The dual graphics cards inside it are not cheap at all. As servers with GPUs are fairly niche, this might actually be a cheaper solution. 3. The form factor. Even if you could create PCs that are cheaper with the same spec, they'll use more power, possibly require more cooling (Mac Pro has a great cooling architecture) and will take up a lot more space. I'd be very interested in hearing how they manage updates and provisioning, however. I can't imagine that'd be much fun on OS X but perhaps there's a way of doing it with OS X Server.

exelius 11y ago
The Mac Pro isn't a great value in the datacenter space. It's a single socket server that's limited to 64 GB of RAM. It's not unusual anymore to throw GPUs in rack mount systems; most of them already have the PCIe bandwidth necessary to support 4 big GPUs so it's often just a matter of getting the right riser cards. Compare a Mac Pro to an HP DL360 that can hold 4 8-core Xeons (32 cores total) and over 200GB of RAM along with a few FirePro or Titan GPGPUs, and the HP will give you far greater density (though a rack mount system with 4 8-core Xeons and 4 GTX Titans would be a power and cooling nightmare!). That said, the Mac Pro isn't as far behind as I would have expected. But OS X also kicks ass at multithreading, especially if you use Apple's graphics libraries. It's entirely possible they get much greater performance from OS X than a Linux or Windows based solution could provide.
- robbyt 11y ago
  OS X does not have NUMA. It has some nice libraries for multi-threading, but that doesn't really matter that much when you're saturating your memory bus because the CPUs are doing too many cross-zone memory requests.
  - exelius 11y ago
    NUMA is irrelevant anyway because there are currently no multi-socket OS X machines. Multiple cores on the same package share a memory controller.
    - deleted 11y ago
      [deleted]
  - acdha 11y ago
    While OS X doesn't have something like Linux's NUMA interface to explicitly lock a thread to a core, 10.5 shipped a thread affinity API which allows you to help the scheduler make better placement decisions: “OS X does not export interfaces that identify processors or control thread placement—explicit thread to processor binding is not supported. Instead, the kernel manages all thread placement. Applications expect that the scheduler will, under most circumstances, run its threads using a good processor placement with respect to cache affinity. However, the application itself knows the detailed caching characteristics of its threads and its data—in particular, the organization of threads as disjoint sets characterized by their association with (affinity to) distinct shared data. While threads within such a set exhibit affinity with each other via shared data, they share a disaffinity or negative affinity with respect to other sets. In other words, a set expresses an affinity with an L2 cache and the scheduler should seek to run threads in a set on processors sharing that L2 cache.” https://developer.apple.com/library/mac/releasenotes/Performance/RN-AffinityAPI/ https://developer.apple.com/library/mac/releasenotes/Perform...
- mozumder 11y ago
  Well, they're fitting 4 sockets and 8 GPUS in the space that's normally used by 4 sockets only. Also, if you're trying to sync raw images between OS X clients and the cloud, then you're going to need OS X servers in the cloud. It'll greatly complicate the clients workflow if they can't use their built in raw converters.
  - exelius 11y ago
    Right; which is why I was surprised that OS X stacks up as favorably as it does. But I wouldn't call it a great value since it's still much more expensive than a traditional rack mount setup. From a density perspective it's really not bad at all, which is a testament to how well-engineered the Mac Pro is.
    - mozumder 11y ago
      I'd say it's not even as expensive as something like IBM's 4-socket rack servers like the x3850 X6.
      - exelius 11y ago
        Yeah, except IBM's x86 hardware has always been stupid expensive for no good reason other than it's IBM. And didn't they spin off their xSeries server business to Lenovo once the market settled on HP systems that cost half what IBM's did?
        mozumder 11y ago
        Even an HP 4 socket system like the DL980 G7 is going to be as expensive. None of these servers are going to be cheap.
        rconti 11y ago
        Also the DL980 G7 is an unholy piece of crap. HP doesn't know how to build or fix them. I've gone through countless service requests on just a dozen machines or thereabouts. It's the worst piece of any kind of hardware I've ever used, hands down.
- semi-extrinsic 11y ago
  No sane person is putting GTX cards into a configuration with that level of power density, you'd have reliability issues from day one. This use case is exactly why Nvidia makes Tesla cards.
- lectrick 11y ago
  The manager of the datacenter kicked in a comment 9 minutes ago or so defending the decision (with facts) at the same level of your comment, if you want to check that out.
skuhn 11y ago
(I'm the datacenter manager at imgix, and I wrote this article) 1. Yeah, the OS X graphics pipeline is at the heart of our desire to use Macs in production. It's also pretty sweet to be able to prototype features in Quartz Composer, and use this whole ecosystem of tools that straight up don't exist on Linux. 2. I mentioned this elsewhere already, but it is actually a pretty good value. The chassis itself is not a terrible expense, and it's totally passive. It really boils down to the fact that we want to use OS X, and the Mac Pros are the best value per gflop in Apple's lineup. They're also still a good value when compared against conventional servers with GPUs, although they do have some drawbacks. 3. I would love it if they weren't little cylinders, but they do seem to handle cooling quite well. The power draw related to cooling for this rack versus a rack of conventional servers is about 1-5/th to 1/10th as much. In terms of provisioning, we're currently using OS X Server's NetRestore functionality to deploy the OS. It's on my to-do list to replicate this functionality on Linux, which should be possible. You can supposedly make ISC DHCPd behave like a BSDP server sufficiently to interoperate with the Mac's EFI loader. We don't generally do software updates in-place, we just reinstall to a new image. However, we have occasionally upgraded OS X versions, which can be done with CLI utilities.
- kaolinite 11y ago
  Thanks for replying and thanks for the article too - great read with some fantastic photography. Really interesting to hear how you provision servers, had no idea that OS X Server came with tools for that, but it certainly makes sense. I wouldn't have thought Apple would have put much time or thought into creating tools for large deployments, but glad to hear that they have.
  - skuhn 11y ago
    Thanks, the photography was done by our lead designer, Miguel. I am super impressed at what he's been able to capture in an environment that can easily come off as utilitarian and sterile. He has some other work online that you might enjoy, not related to Macs or imgix: http://photos.miggi.me/ http://photos.miggi.me/
- chiph 11y ago
jseliger 11y ago
2. The Mac Pro, whilst expensive, is good value for money. The dual graphics cards inside it are not cheap at all. As servers with GPUs are fairly niche, this might actually be a cheaper solution. I'd actually qualify this ever-so-slightly by saying "It's a good value for money if you need the specific features it offers." Which it evidently does to the OP! But many of us would prefer something with, say, one video card, one mainstream-ish desktop processor, and one mechanical hard drive, an way lower costs.
- skuhn 11y ago
  Yes, you're absolutely right. We use conventional Linux servers for application, database and storage for exactly this reason. The Mac Pros are a good value for an image rendering machine, but not for general purpose server stuff (in my opinion). It's also a bit dear for use as a desktop machine, but it is pretty nice to have one hanging out for on your desk for a few weeks.
sanjeetsuhag 11y ago
From the site : "Building on OS X technologies means we’re dependent on Apple hardware for this part of the service, but we aren’t necessarily limited to Mac Minis. Apple’s redesigned Mac Pro seemed like an ideal replacement, as long as we could reliably operate it in a datacenter environment."

exelius 11y ago

The Mac Pro isn't a great value in the datacenter space. It's a single socket server that's limited to 64 GB of RAM. It's not unusual anymore to throw GPUs in rack mount systems; most of them already have the PCIe bandwidth necessary to support 4 big GPUs so it's often just a matter of getting the right riser cards. Compare a Mac Pro to an HP DL360 that can hold 4 8-core Xeons (32 cores total) and over 200GB of RAM along with a few FirePro or Titan GPGPUs, and the HP will give you far greater density (though a rack mount system with 4 8-core Xeons and 4 GTX Titans would be a power and cooling nightmare!). That said, the Mac Pro isn't as far behind as I would have expected. But OS X also kicks ass at multithreading, especially if you use Apple's graphics libraries. It's entirely possible they get much greater performance from OS X than a Linux or Windows based solution could provide.

robbyt 11y ago
OS X does not have NUMA. It has some nice libraries for multi-threading, but that doesn't really matter that much when you're saturating your memory bus because the CPUs are doing too many cross-zone memory requests.
- exelius 11y ago
  NUMA is irrelevant anyway because there are currently no multi-socket OS X machines. Multiple cores on the same package share a memory controller.
  - deleted 11y ago
    [deleted]
- acdha 11y ago
  While OS X doesn't have something like Linux's NUMA interface to explicitly lock a thread to a core, 10.5 shipped a thread affinity API which allows you to help the scheduler make better placement decisions: “OS X does not export interfaces that identify processors or control thread placement—explicit thread to processor binding is not supported. Instead, the kernel manages all thread placement. Applications expect that the scheduler will, under most circumstances, run its threads using a good processor placement with respect to cache affinity. However, the application itself knows the detailed caching characteristics of its threads and its data—in particular, the organization of threads as disjoint sets characterized by their association with (affinity to) distinct shared data. While threads within such a set exhibit affinity with each other via shared data, they share a disaffinity or negative affinity with respect to other sets. In other words, a set expresses an affinity with an L2 cache and the scheduler should seek to run threads in a set on processors sharing that L2 cache.” https://developer.apple.com/library/mac/releasenotes/Performance/RN-AffinityAPI/ https://developer.apple.com/library/mac/releasenotes/Perform...
mozumder 11y ago
Well, they're fitting 4 sockets and 8 GPUS in the space that's normally used by 4 sockets only. Also, if you're trying to sync raw images between OS X clients and the cloud, then you're going to need OS X servers in the cloud. It'll greatly complicate the clients workflow if they can't use their built in raw converters.
- exelius 11y ago
  Right; which is why I was surprised that OS X stacks up as favorably as it does. But I wouldn't call it a great value since it's still much more expensive than a traditional rack mount setup. From a density perspective it's really not bad at all, which is a testament to how well-engineered the Mac Pro is.
  - mozumder 11y ago
    I'd say it's not even as expensive as something like IBM's 4-socket rack servers like the x3850 X6.
    - exelius 11y ago
      Yeah, except IBM's x86 hardware has always been stupid expensive for no good reason other than it's IBM. And didn't they spin off their xSeries server business to Lenovo once the market settled on HP systems that cost half what IBM's did?
      - mozumder 11y ago
        Even an HP 4 socket system like the DL980 G7 is going to be as expensive. None of these servers are going to be cheap.
        rconti 11y ago
        Also the DL980 G7 is an unholy piece of crap. HP doesn't know how to build or fix them. I've gone through countless service requests on just a dozen machines or thereabouts. It's the worst piece of any kind of hardware I've ever used, hands down.
semi-extrinsic 11y ago
No sane person is putting GTX cards into a configuration with that level of power density, you'd have reliability issues from day one. This use case is exactly why Nvidia makes Tesla cards.
lectrick 11y ago
The manager of the datacenter kicked in a comment 9 minutes ago or so defending the decision (with facts) at the same level of your comment, if you want to check that out.

skuhn 11y ago

(I'm the datacenter manager at imgix, and I wrote this article) 1. Yeah, the OS X graphics pipeline is at the heart of our desire to use Macs in production. It's also pretty sweet to be able to prototype features in Quartz Composer, and use this whole ecosystem of tools that straight up don't exist on Linux. 2. I mentioned this elsewhere already, but it is actually a pretty good value. The chassis itself is not a terrible expense, and it's totally passive. It really boils down to the fact that we want to use OS X, and the Mac Pros are the best value per gflop in Apple's lineup. They're also still a good value when compared against conventional servers with GPUs, although they do have some drawbacks. 3. I would love it if they weren't little cylinders, but they do seem to handle cooling quite well. The power draw related to cooling for this rack versus a rack of conventional servers is about 1-5/th to 1/10th as much. In terms of provisioning, we're currently using OS X Server's NetRestore functionality to deploy the OS. It's on my to-do list to replicate this functionality on Linux, which should be possible. You can supposedly make ISC DHCPd behave like a BSDP server sufficiently to interoperate with the Mac's EFI loader. We don't generally do software updates in-place, we just reinstall to a new image. However, we have occasionally upgraded OS X versions, which can be done with CLI utilities.

kaolinite 11y ago
Thanks for replying and thanks for the article too - great read with some fantastic photography. Really interesting to hear how you provision servers, had no idea that OS X Server came with tools for that, but it certainly makes sense. I wouldn't have thought Apple would have put much time or thought into creating tools for large deployments, but glad to hear that they have.
- skuhn 11y ago
  Thanks, the photography was done by our lead designer, Miguel. I am super impressed at what he's been able to capture in an environment that can easily come off as utilitarian and sterile. He has some other work online that you might enjoy, not related to Macs or imgix: http://photos.miggi.me/ http://photos.miggi.me/
chiph 11y ago

jseliger 11y ago

2. The Mac Pro, whilst expensive, is good value for money. The dual graphics cards inside it are not cheap at all. As servers with GPUs are fairly niche, this might actually be a cheaper solution. I'd actually qualify this ever-so-slightly by saying "It's a good value for money if you need the specific features it offers." Which it evidently does to the OP! But many of us would prefer something with, say, one video card, one mainstream-ish desktop processor, and one mechanical hard drive, an way lower costs.

skuhn 11y ago
Yes, you're absolutely right. We use conventional Linux servers for application, database and storage for exactly this reason. The Mac Pros are a good value for an image rendering machine, but not for general purpose server stuff (in my opinion). It's also a bit dear for use as a desktop machine, but it is pretty nice to have one hanging out for on your desk for a few weeks.

sanjeetsuhag 11y ago

From the site : "Building on OS X technologies means we’re dependent on Apple hardware for this part of the service, but we aren’t necessarily limited to Mac Minis. Apple’s redesigned Mac Pro seemed like an ideal replacement, as long as we could reliably operate it in a datacenter environment."