5 ms·

(I'm the datacenter manager at imgix, and I wrote the article) I've alluded to this elsewhere, but the math doesn't add up to your gut reaction. It's cheaper,

by skuhn 11y ago

(I'm the datacenter manager at imgix, and I wrote the article) I've alluded to this elsewhere, but the math doesn't add up to your gut reaction. It's cheaper, but not by a significant enough margin relative to the engineering costs, to go with commodity servers and GPUs. Building things out of sheet metal is actually easier than migrating to Linux, for one big reason: we can pay someone else to do it, because it isn't part of our core competency. In fact, I'm pushing to open source the design of this chassis, in tandem with our design partner (Racklive). Not sure if it will happen, but I'd love to see it.

x0054 11y ago
It strikes me that OS X is a solid server platform. Linux is always a more flexible choice, but if OS X works for you, it works for you. That being said, is sticking cylindrical mac pros on their side into square racks really the best solution. There are 2 problems I see with this design: 1: You are placing the Mac Pros on their side, which may lead to premature bearing failure on the main cooling fan. Apple designed the cooling fan to be as silent as possible, which means that they optimized the bearing and the fan to work in vertical orientation. Bearings designed for thrust (vertical) orientation may not work so well if placed horizontally for a long time. 2: You are fitting triangular shaped computers, wrapped into round cases, into square shaped boxes, resulting in significant loss of space density. Considering that Apple is a huge company that owns huge data centers, combined with the fact that it would be simply stupid for a company who makes their own OS to run anything but that OS, and combined with the above mentioned problems with using Mac Pros as server "logs" (because you cant call them blades), I would assume that Apple has internally OSX servers designed in the traditional blade configuration. They may not sell or advertise them, but they MUST have them. Given that you guys are buying a ton of hardware, and are located nearby, and would be actively promoting running Apple hardware, wouldn't it be wise to at least approach Apple and see if they would be kind enough to sell you some of those blade form factor servers they simply must have. I may be completely wrong here, but apple did brag about how Swift is the new language thats so flexible that you can make a Mobile app in it, or a Desktop app, or even a full blown social network. If that's the case, they must have some plans for the server market? No? Any way, in the end it's a cool design, but I would seriously consider at least stacking the Mac Pros vertically to avoid fan issues. You can actually get a tighter form factor that way as well, unless space is not the issue. And if it's not, then hell, what's wrong with just placing a bunch of Pros on an Ikea shelf in a well air-conditioned room :)
- skuhn 11y ago
  1. That's certainly a possibility, and one that we won't really have hard numbers on for some time to come. However, the fan is about a $60 part, so provided that we don't have coordinated, catastrophic failures and that they live for at least a year, we're doing alright. Do note that Apple specifically says that the Mac Pro may be operated on its side. https://support.apple.com/en-us/HT201379 https://support.apple.com/en-us/HT201379 2. True, but 1U per server is not bad density by any stretch. For my app servers, they effectively occupy 0.5U; database and storage effectively occupy 1U. So this puts the Mac Pros on par with the larger server class. Were we to deploy renderers in conventional server chassis, a similar system would occupy at least an effective 1U if not a full 2U. What Apple does internally is, of course, shrouded in mystery. I know some people there, and we talk to people when we can, but they just aren't the kind of company that is going to tell you how they make the sausage. From what I've heard and my sense from speaking with them over the years, they do not use OS X in production. They used to use Darwin and Solaris, and now almost exclusively use Linux (presumably Solaris is still around to run Oracle). They did used to use Xserves internally, but even at their scale it isn't worth building them just for their own use.
  - x0054 11y ago
    Fascinating, I had no idea Apple approves using the Mac Pros on their side. It would be interesting to find out what happens with the fans. It's also fascinating that they are running Linux internally nowadays, for their server side stuff. What next, I find out that all of the Microsoft data centers run Debian :) Considering that they employ all of those Objective-C and Swift engineers, you would thing that they would want to leverage their workforce write Obj-C or Swift backend code as well. For most backend tasks either Swift or Obj-C is as good of a language as any other. Any way, rackable OS X systems are a missed opportunity for Apple. They can sell them to a company like yours, movie production houses, and even design some libraries and make a play for the web app market with Swift. Not sure how successful the last one would be. As for the economies of scale, they don't even need to manufacture or design the system, take an off the shelf rack mount server from another manufacturer, fiddle around with the casing a bit to give it that Apple feel, and load OSX on it. Perhaps the margins in the server side hardware are way too slim.

by skuhn 11y ago

x0054 11y ago

It strikes me that OS X is a solid server platform. Linux is always a more flexible choice, but if OS X works for you, it works for you. That being said, is sticking cylindrical mac pros on their side into square racks really the best solution. There are 2 problems I see with this design: 1: You are placing the Mac Pros on their side, which may lead to premature bearing failure on the main cooling fan. Apple designed the cooling fan to be as silent as possible, which means that they optimized the bearing and the fan to work in vertical orientation. Bearings designed for thrust (vertical) orientation may not work so well if placed horizontally for a long time. 2: You are fitting triangular shaped computers, wrapped into round cases, into square shaped boxes, resulting in significant loss of space density. Considering that Apple is a huge company that owns huge data centers, combined with the fact that it would be simply stupid for a company who makes their own OS to run anything but that OS, and combined with the above mentioned problems with using Mac Pros as server "logs" (because you cant call them blades), I would assume that Apple has internally OSX servers designed in the traditional blade configuration. They may not sell or advertise them, but they MUST have them. Given that you guys are buying a ton of hardware, and are located nearby, and would be actively promoting running Apple hardware, wouldn't it be wise to at least approach Apple and see if they would be kind enough to sell you some of those blade form factor servers they simply must have. I may be completely wrong here, but apple did brag about how Swift is the new language thats so flexible that you can make a Mobile app in it, or a Desktop app, or even a full blown social network. If that's the case, they must have some plans for the server market? No? Any way, in the end it's a cool design, but I would seriously consider at least stacking the Mac Pros vertically to avoid fan issues. You can actually get a tighter form factor that way as well, unless space is not the issue. And if it's not, then hell, what's wrong with just placing a bunch of Pros on an Ikea shelf in a well air-conditioned room :)

skuhn 11y ago
1. That's certainly a possibility, and one that we won't really have hard numbers on for some time to come. However, the fan is about a $60 part, so provided that we don't have coordinated, catastrophic failures and that they live for at least a year, we're doing alright. Do note that Apple specifically says that the Mac Pro may be operated on its side. https://support.apple.com/en-us/HT201379 https://support.apple.com/en-us/HT201379 2. True, but 1U per server is not bad density by any stretch. For my app servers, they effectively occupy 0.5U; database and storage effectively occupy 1U. So this puts the Mac Pros on par with the larger server class. Were we to deploy renderers in conventional server chassis, a similar system would occupy at least an effective 1U if not a full 2U. What Apple does internally is, of course, shrouded in mystery. I know some people there, and we talk to people when we can, but they just aren't the kind of company that is going to tell you how they make the sausage. From what I've heard and my sense from speaking with them over the years, they do not use OS X in production. They used to use Darwin and Solaris, and now almost exclusively use Linux (presumably Solaris is still around to run Oracle). They did used to use Xserves internally, but even at their scale it isn't worth building them just for their own use.
- x0054 11y ago
  Fascinating, I had no idea Apple approves using the Mac Pros on their side. It would be interesting to find out what happens with the fans. It's also fascinating that they are running Linux internally nowadays, for their server side stuff. What next, I find out that all of the Microsoft data centers run Debian :) Considering that they employ all of those Objective-C and Swift engineers, you would thing that they would want to leverage their workforce write Obj-C or Swift backend code as well. For most backend tasks either Swift or Obj-C is as good of a language as any other. Any way, rackable OS X systems are a missed opportunity for Apple. They can sell them to a company like yours, movie production houses, and even design some libraries and make a play for the web app market with Swift. Not sure how successful the last one would be. As for the economies of scale, they don't even need to manufacture or design the system, take an off the shelf rack mount server from another manufacturer, fiddle around with the casing a bit to give it that Apple feel, and load OSX on it. Perhaps the margins in the server side hardware are way too slim.