9 ms·

160 Mac Minis, One Rack

214 points by barredo 14y ago

alanctgardner2 14y ago
It's very cool, but is anyone actually tied to OS X as a server platform? Couldn't they move to FreeBSD and save a ton of money in an application like this? I'm wondering if there's a real business case for this, or it's just a fun hack. edit: I guess lumped into this is the small market that seems to exist for colocated Mac Minis. Is there something about them that is better than renting commodity x64 hardware?
- ahi 14y ago
  One of my old employers has legacy systems built with WebObjects. In theory, WebObjects is Java and should run on anything, but typically it is on OS X Server. I left just as one of my colleagues was trying to figure out what to do about the death of the XServe.
- nbpoole 14y ago
  First idea that comes to mind: http://stackoverflow.com/questions/3837043/best-practice-for-setting-up-an-automated-build-server-for-iphone-apps http://stackoverflow.com/questions/3837043/best-practice-for...
- chrismealy 14y ago
  In comments the builder said it's for testing.
- taligent 14y ago
  I am currently using a cluster of Mac Minis as a server platform. Some benefits: * Launchd is a massive improvement over the equivalent mess on Linux. This can't be understated if you are managing your own hardware. * You can develop on the same machine you are deploying to. * You have exactly the same toolchain as on Linux. * Lots of remote monitoring options that are unique to OSX e.g. OSX Server * The OS is stable and upgrades are safe enough to enable auto update. I could never do that on CentOS. But really it comes down to hardware and resale value for me. 2 Mac Minis in 1RU is great value.
  - cheald 14y ago
    I don't have extensive experience with Launchd, but what makes it better? OS X doesn't have the same toolchain as Linux - notably, Apple doesn't ship anything licensed under GPLv3. This leads to OS X versions of common GNU tools being extremely outdated. I do agree with the point about developing and deploying in the same environment. I do both in Linux for that exact reason; far fewer surprises when your deploy environment matches what you've already debugged.
    - taligent 14y ago
      Everything. It is all in one place, very simple to use, has process monitoring like Monit and well supported. Nobody uses what Apple ships by default. Homebrew or MacPorts and I have exactly the same setup as I have on my Linux server. Plus lots of amazing GUI wrappers which can simplify setup and ongoing maintenance.
      - chimeracoder 14y ago
        >Homebrew or MacPorts and I have exactly the same setup as I have on my Linux server. My biggest problems with the above: 1. Neither Homebrew nor MacPorts (nor fink, for that matter) has a complete set of relevant packages 2. Compilation from source is ridiculously slow. 3. Certain utilities are best installed from .dmg files instead, and it's a pain to remember which was installed which way.
        rymith 14y ago
        1. Name your package manager, and I can list the missing packages, or missing flags which have made my life a pain at some point in my career. 2. Yup, and the only way, should you need to set any flags. 3. True. Which is why it's nice that you can mount dmg's and install pkg's via the command line.
        chimeracoder 14y ago
        > 2. Yup, and the only way, should you need to set any flags. Man, if I subscribed to that belief, I might as well just use Gentoo!
- mminer 14y ago
  It's a niche use case, but Apple's Qmaster software facilitates distributed video rendering and exporting for Final Cut Pro and some of their other pro applications. A server cluster like this would probably be overkill, but when you're working with massive video files, extra (OS X) machines to do the heavy lifting make a huge difference.
- hugs 14y ago
  > anyone actually tied to OS X as a server platform? Yes, if you make software that runs on OS X (or iOS and you want to test on iOS Simulators) you need OS X machines for your build and test process. You need lots of machines so you and your fellow developers can run speed things up and run tests in parallel.
madao 14y ago
Considering you only really have to pay around the 60 dollar mark for the OS now, I dont think its much of a big deal, I use one of these at home as a mini fileserver/wiki it draws sweet FA makes little to no noise and has HDMI connector direct into my tv. I would happily deploy one for our company marketing team or small scale offices.
- dfc 14y ago
  "Draws sweet FA"?
  - alanctgardner2 14y ago
    FA = "Fuck All" It draws precious little electricity, in other words.
    - madao 14y ago
      Sorry, my Aussie is showing again...
      - mitchty 14y ago
        No worries, I'm american and caught on.
    - akent 14y ago
      The more polite version is "Sweet Fannie Adams". http://www.phrases.org.uk/meanings/sweet-fanny-adams.html http://www.phrases.org.uk/meanings/sweet-fanny-adams.html
dreadsword 14y ago
Seems like an expensive way to do data center stuff. Why not create a rack full of alienware laptops or something?
- taligent 14y ago
  Actually it's far cheaper than even an equivalent Supermicro solution let alone HP/Dell etc. You are getting at minimum 2 Mac Minis in 1RU which as of today could be a 8 core Core i7 / Dual SSDs / 16GB RAM. Plus if you want to upgrade them then you can put them on eBay and get 75% of the original cost back. Try doing that with a server.
  - gonzo 14y ago
    The machine you quote (Mac mini: 8 core i7 / Dual SSD / 16GB ram) does not exist. The closest thing to it (Mac mini: 4 core i7 / single SSD / 16GB ram) costs $1499 from Apple. Are you proposing that I can't get a dual i7 @ 2.6GHz, dual SSD, 16GB ram from Supermicro for less than $3,000? I have a mini on my desk, but even as an Apple fan, I think you're off the mark here.
    - taligent 14y ago
      Mac mini is hyper threaded so appears in the OS as 8-cores. The OSX Server edition starts at $999. Replace the two hard drives with two SSD ($200) and add 16GB RAM ($100). Now show me a Supermicro that will have two nodes in 1RU for equivalent price. Then remind me how much it is going to sell in a year when I decide to upgrade. I guarantee that the Mac Minis would sell in a day on eBay.
      - theatrus2 14y ago
        You're not going to beat the two-nodes price. If you're fine with a single-NIC single-disk server, Minis are not a bad deal. However, you're not going to get dual-NICs, ECC memory, or multiple "spindles" in a Mini formfactor.
gonzo 14y ago
it was when he started talking about using solder on a 220 VAC connection that I lost the faith in him knowing how to do it right.
- wlesieutre 14y ago
  Out of curiosity, what's the problem with that? I assume he'd have the copper twisted together to make a good connection, with the solder just holding it in place instead of acting as the conductor.
  - zalzane 14y ago
    depending on the resistivity of the copper wire versus the lead-tin mix in the solder, there may end up being more current going through the solder than the actual wire. Even if the resistivity values of the copper and solder are similar (I'm pretty sure they're at least in the same order of magnitude), you'll still end up with the same amount of current running through the copper wire and the solder.
    - Tloewald 14y ago
      Um, what? You'll dissipate more heat at 110v than 220v (high school physics -- heat dissipation is proportional to resistance and the square of current, so for a given wattage, it is inversely proportional to voltage — hence high voltage power transmission). In any event, 220v is no big deal. It's household current almost everywhere except the US. (A few countries use 110V, Australia is supposedly 240v.)
      - zalzane 14y ago
        The heat dissipated of the two metals would be identical if their resistances were the same, of course. I'm not even considering the difference in amperage between a 110V and 220V connection, as I'm pretty sure that's not what the parent parent post was in reference to. I was under the impression that the issue at hand was that using solder for high-load wires was bad because of solder's extremely low melting point. I meant to point out that even though the solder is just holding the wires together, there is still a current running through it, and if the resistivity of the solder is lower than that of the copper, the chances of the solder melting would be quite a bit higher. I honestly don't know if that's why the parent parent post mentioned that soldering high load wires are bad. I just extrapolated issues that someone might have when using solder on high load wires, and relayed them in my reply.
tzaman 14y ago
I'm a proud owner of a Mini Server (slightly customised - replaced memory and primary disk with SSD) for over a year. I use it as my main workstation and I love it; So small (and relatively cheap including the upgrade), yet so powerful.
jurre 14y ago
Before the thread becomes cluttered with people suggesting alternatives or questioning why you wouldn't just run <insert manufacturer, OS, etc>, the person that did this replied the following: simbimbo says: December 9, 2012 at 11:03 am Thanks for the great write up Hack A Day. I would like to answer some of the questions posted. @Geebles these machines all run SSD’s and I ordered them with AppleCare, so I hope to never have to change a drive ;-) As for the reason I built this.. Well, I guess I’m just like a challenge ;-), but seriously, the company I work for has a need to have large numbers of machines to build and test the software we make. There were plenty of discussions of Virtual environments and other “Bare Motherboard”/Google Datacenter-type solutions, but the fact is, the Apple EULA requires that Mac OS X run on Apple Hardware, since we are a software company we adhere to these rules without exception. These Mac Machines all run OS X in a NetBooted environment. We require Mac OS X because the products we make support Windows, Linux and Mac so we have data centers with thousands of machines configured with all 3 OS’s running constant build and test operations 24 hours a day 365 days a year. As for device failure, we treat these machines like pixels in a very large display, if a few fail, it’s ok, the management software disables them until we can switch them out. This approach allows us to continue our operations regardless of machine failures. @bitbass I tried the vertical approach, but manufacturing the required plenum to keep the air clean to the rear machines cost too much for this project, but it’s not off the table for the next rack @Kris Lee When I open the door I can literally watch the machine temps go up, but I can keep it open for 15-20 minutes before the core temps reach 180F @Adam Ahhh.. Nope, you can’t have my job ;-)
- shardling 14y ago
  I believe Mozilla has large racks of mac Minis for testing Firefox on. (Same issue, I believe -- to support multiple platforms on the same hardware, you need something that's legal to run OSX on.) e: Ah, according to [this article](http://www.wired.com/wiredenterprise/2012/05/mozillas-new-datacenter/ http://www.wired.com/wiredenterprise/2012/05/mozillas-new-da...) they have 500 Minis in their data center.
  - deleted 14y ago
    [deleted]
frozenport 14y ago
Why not use 2 racks? This would solve his thermal dissipation problem and probably be easier when compared to getting custom hardware.
- liquidise 14y ago
  i would guess floor real estate constraints. Why do in 2 racks what you can do cost effectively in 1?
  - keypusher 14y ago
    Because it's going to be extremely hot.
- simbimbo 14y ago
  Rack "foorprints" at the datacenter are expensive, and you pay for 15kw per footprint whether you use it or not. it just made sense to fit it all into one. Having had this rack running at full power for a couple of weeks now, I can say the temps stay lower than our SuperMicro racks in the same row!! and the Supermicro racks can only hold 20 machines before they run out of power in the footprint.
meaty 14y ago
I'm actually surprised any DC would take that equipment. They, in my experience at least, are very fussy about what what you put in the racks and power draw etc. Oh and we get 640 cores in 20U (8x4 core xeon machines each 1u) and that leaves enough room for a 32Tb SAN, FC switches and a pair of redundant LAN switches. REgarding splitting the power using the hack described, 160 melted minis and a halon cloud coming up. Looks pretty though.
- jws 14y ago
  REgarding splitting the power using the hack described, 160 melted minis and a halon cloud coming up. You should have a talk with your power cord provider. You should be using cables that can handle at least 4 amps in anything with a 110v plug on it. I don't think you can buy one smaller than 18ga and those are good for 10 amps. Remember, you have to handle enough current to blow the breaker if something goes wrong (unless you are British and have your own fuse in the plug).
  - simbimbo 14y ago
    We went through several designs with the vendor. This cable is designed to operate at the circuit rating. The wire from the PDU plug to the split is 12GA 600V and the cable from the split to the mini is 14GA 300V.
- michaelt 14y ago
  A 1U xserve maxes out at 334 Watts [1]. Each Mac mini takes 85 Watts [2], so 4 is 340 Watts. At 220v that's 1.54 Amps. Is splitting the power really so dangerous? [1] http://support.apple.com/kb/HT3513 http://support.apple.com/kb/HT3513 [2] http://support.apple.com/kb/HT3468 http://support.apple.com/kb/HT3468
- alexkus 14y ago
  It must be their own DC. Most rackspace rented (in UK DC's at least) tend to be a maximum of 16A (at 240V) per 42U cabinet, so just under 4kW. By my estimation those Mac Minis will be drawing ~13kW at peak.
w3pm 14y ago
I understand the idea of treating them like pixels, so if a fan dies or a NIC card dies, no problem, just stop using that Mini. But what about memory corruption or other issues that are more difficult to detect? Normally server hardware has things like ECC memory to prevent these issues, but in this case a Mini with bad RAM could intermittently corrupt data for some time before it's noticed (if ever).
- lallysingh 14y ago
  The machines are for testing. They'll detect those through secondary means. If a machine's faulty, it'll cause two cases: (1) faulty software will register as faulty; (2) good software will register as faulty. The third case (faulty software marked as good), is really unlikely, and any time it does happen, a later bug report will give a hint. A test failure will probably bring up an engineer that will track down the issue, and a re-test will inevitably occur. The faulty machine will eventually (hopefully) get labeled flaky and will get repaired. Of course, nobody may care and just use a double-test to verify that an executable is good.
  - rdtsc 14y ago
    > A test failure will probably bring up an engineer that will track down the issue, and a re-test will inevitably occur. The faulty machine will eventually (hopefully) get labeled flaky and will get repaired. Depending on how valuable the engineers' time is. I have seen this played out like this: hardware gets blamed last after hours and days of testing have been wasted. So tests are run and re-run, blame goes all around until finally after hours and hours of testing it is determined that maybe it is hardware after all. In the end an engineers' time is worth a lot more than savings obtained by running flaky but cheaper hardware.
rdtsc 14y ago
Mac Minis are horrible server hardware. We've had a couple running as servers. They fail randomly. Their hard drives fail. They don't rack mount easily. The only reason to have them is if you inherit some old ones, don't want to throw them away, and then don't mind replacing and throwing failed units away pretty often.
- sliverstorm 14y ago
  Well, they do provide good machine density when you rack them like he did.
  - rdtsc 14y ago
    Yap definitely true.
- taligent 14y ago
  All servers fail randomly and so do hard drives so not sure why Mac Mini would be any different. Also they rack mount very easily: http://www.sonnettech.com/product/rackmacmini.html http://www.sonnettech.com/product/rackmacmini.html
  - rdtsc 14y ago
    They do but we found that mac minis do so more often. I have a sample of about 15 out of say 100+ other regular 1U rack mountable machines. Mac minis failed at a much higher rate. The worst part is that some failures were not a 'stop dead failure' where it was easy to detect and replace. Some would just freeze intermittently, some just got really slow. One could tell that there is a practical difference between server grade and consumer grade components. In retrospect time spent debugging and messing with this was probably not worth the gain in density and the cost.
    - adrianpike 14y ago
      I for one would love it if you'd have the chance to dig a little bit and share what you find. I've only got a sample size of two recently purchased (2012) Mini's, and they've both had memory issues.
datums 14y ago
Definitely a fun challenge. If you're going to invest in the hardware and custom build. Forget the y cable. Figure out a better solution. Rent 1/2 rack next to it to hold the pdus. +1 on the massive door fans.
patangay 14y ago
We did something similar at Facebook for iOS and OSX automated testing and a few of them doing iOS app builds. Here is a post that Jay Parikh (VP of Infrastructure) made about it. http://tinyurl.com/cnvss4v http://tinyurl.com/cnvss4v Our density isn't as high (we have 64 minis) because of cooling and cabling that we designed according to our datacenter cooling standards. @jurre - If you want to chat about our design, message me and I can put you in touch with our hardware designer.
- simbimbo 14y ago
  I would like to chat about it.
- i386 14y ago
  Would love to hear more about your automated testing!
- jurre 14y ago
  I just copied that reply from the person that built the actual rack, but I would love to hear more about your iOS testing infrastructure/process!
  - jevinskie 14y ago
    Me too! Before we used fruitstrap[0] we used DeviceAnywhere. This was a 2U rack unit that contained a franken-iPhone. The hardware buttons were soldered up to GPIO on the DeviceAnywhere. [0]: https://github.com/ghughes/fruitstrap https://github.com/ghughes/fruitstrap
- patangay 14y ago
  For those who showed interest in learning more about our infrastructure/automated testing process, if you could drop me an email? gp at our corp domain fb.com. I don't work on the team anymore, but I can probably start off a thread with the right people involved from Facebook's side.
j45 14y ago
I would love to get the specs on the tray that he put the 4 mac mini's on.
- simbimbo 14y ago
  They're in production now.. H-Squared. great vendor. http://h-sq.com/products/minirack/index.html http://h-sq.com/products/minirack/index.html
  - j45 14y ago
    Great, thanks for the pointer!
georgebarnett 14y ago
Interestingly, it looks like the front fans blow _into_ the rack. This means that if the door isn't securely closed it'll blow open - being on hinges and having massive fans attached. It would be better to have the fans on the back and suck air through the rack rather. That said, DC floor space is cheap compared to power and cooling. I'm surprised they didn't lower the density so as not to have a massive fire risk.
- simbimbo 14y ago
  I looked into some rack cooling options, but was unable to find a solution that would provide the amount and wide coverage of airflow I needed to move air slowly and uniformly through the rack to provide effective cooling. ( I was designing this to be used with the active cooling rear doors, so I couldn't overwhelm the door with too much air or it wouldn't cool the air effectively, and would raise the ambient temp of the room). So the fans move a high volume of air through the entire cabinet (including the corners) at low velocity resulting in very effective cooling of each of the 40 shelves. The fans are large so they move a high volume of air at a low speed, the door doesn't move is left unlatched. Also, can you please explain your "Massive Fire risk" comment? All of the hardware installed in this rack is UL certified and all of the machines will simply shut down if they get too hot.
lallysingh 14y ago
I'm surprised both that the high density worked for consumer devices, and that the rack wasn't prettier.
mewmoo 14y ago
Yes.. but why?
nsxwolf 14y ago
Curious - what do we call a computer like this? It's obviously not going to make the TOP500, but is it a "supercomputer"? I thought perhaps "minisupercomputer" might be fitting, but according to Wikipedia that is a term for a class of computers that became obsolete in the early 90s.
sproketboy 14y ago
Yeah but can it run Crysis? /s

214 points by barredo 14y ago

It's very cool, but is anyone actually tied to OS X as a server platform? Couldn't they move to FreeBSD and save a ton of money in an application like this? I'm wondering if there's a real business case for this, or it's just a fun hack. edit: I guess lumped into this is the small market that seems to exist for colocated Mac Minis. Is there something about them that is better than renting commodity x64 hardware?

ahi 14y ago
One of my old employers has legacy systems built with WebObjects. In theory, WebObjects is Java and should run on anything, but typically it is on OS X Server. I left just as one of my colleagues was trying to figure out what to do about the death of the XServe.
nbpoole 14y ago
First idea that comes to mind: http://stackoverflow.com/questions/3837043/best-practice-for-setting-up-an-automated-build-server-for-iphone-apps http://stackoverflow.com/questions/3837043/best-practice-for...
chrismealy 14y ago
In comments the builder said it's for testing.
taligent 14y ago
I am currently using a cluster of Mac Minis as a server platform. Some benefits: * Launchd is a massive improvement over the equivalent mess on Linux. This can't be understated if you are managing your own hardware. * You can develop on the same machine you are deploying to. * You have exactly the same toolchain as on Linux. * Lots of remote monitoring options that are unique to OSX e.g. OSX Server * The OS is stable and upgrades are safe enough to enable auto update. I could never do that on CentOS. But really it comes down to hardware and resale value for me. 2 Mac Minis in 1RU is great value.
- cheald 14y ago
  I don't have extensive experience with Launchd, but what makes it better? OS X doesn't have the same toolchain as Linux - notably, Apple doesn't ship anything licensed under GPLv3. This leads to OS X versions of common GNU tools being extremely outdated. I do agree with the point about developing and deploying in the same environment. I do both in Linux for that exact reason; far fewer surprises when your deploy environment matches what you've already debugged.
  - taligent 14y ago
    Everything. It is all in one place, very simple to use, has process monitoring like Monit and well supported. Nobody uses what Apple ships by default. Homebrew or MacPorts and I have exactly the same setup as I have on my Linux server. Plus lots of amazing GUI wrappers which can simplify setup and ongoing maintenance.
    - chimeracoder 14y ago
      >Homebrew or MacPorts and I have exactly the same setup as I have on my Linux server. My biggest problems with the above: 1. Neither Homebrew nor MacPorts (nor fink, for that matter) has a complete set of relevant packages 2. Compilation from source is ridiculously slow. 3. Certain utilities are best installed from .dmg files instead, and it's a pain to remember which was installed which way.
      - rymith 14y ago
        1. Name your package manager, and I can list the missing packages, or missing flags which have made my life a pain at some point in my career. 2. Yup, and the only way, should you need to set any flags. 3. True. Which is why it's nice that you can mount dmg's and install pkg's via the command line.
        chimeracoder 14y ago
        > 2. Yup, and the only way, should you need to set any flags. Man, if I subscribed to that belief, I might as well just use Gentoo!
mminer 14y ago
It's a niche use case, but Apple's Qmaster software facilitates distributed video rendering and exporting for Final Cut Pro and some of their other pro applications. A server cluster like this would probably be overkill, but when you're working with massive video files, extra (OS X) machines to do the heavy lifting make a huge difference.
hugs 14y ago
> anyone actually tied to OS X as a server platform? Yes, if you make software that runs on OS X (or iOS and you want to test on iOS Simulators) you need OS X machines for your build and test process. You need lots of machines so you and your fellow developers can run speed things up and run tests in parallel.

madao 14y ago

Considering you only really have to pay around the 60 dollar mark for the OS now, I dont think its much of a big deal, I use one of these at home as a mini fileserver/wiki it draws sweet FA makes little to no noise and has HDMI connector direct into my tv. I would happily deploy one for our company marketing team or small scale offices.

dfc 14y ago
"Draws sweet FA"?
- alanctgardner2 14y ago
  FA = "Fuck All" It draws precious little electricity, in other words.
  - madao 14y ago
    Sorry, my Aussie is showing again...
    - mitchty 14y ago
      No worries, I'm american and caught on.
  - akent 14y ago
    The more polite version is "Sweet Fannie Adams". http://www.phrases.org.uk/meanings/sweet-fanny-adams.html http://www.phrases.org.uk/meanings/sweet-fanny-adams.html

dreadsword 14y ago

Seems like an expensive way to do data center stuff. Why not create a rack full of alienware laptops or something?

taligent 14y ago
Actually it's far cheaper than even an equivalent Supermicro solution let alone HP/Dell etc. You are getting at minimum 2 Mac Minis in 1RU which as of today could be a 8 core Core i7 / Dual SSDs / 16GB RAM. Plus if you want to upgrade them then you can put them on eBay and get 75% of the original cost back. Try doing that with a server.
- gonzo 14y ago
  The machine you quote (Mac mini: 8 core i7 / Dual SSD / 16GB ram) does not exist. The closest thing to it (Mac mini: 4 core i7 / single SSD / 16GB ram) costs $1499 from Apple. Are you proposing that I can't get a dual i7 @ 2.6GHz, dual SSD, 16GB ram from Supermicro for less than $3,000? I have a mini on my desk, but even as an Apple fan, I think you're off the mark here.
  - taligent 14y ago
    Mac mini is hyper threaded so appears in the OS as 8-cores. The OSX Server edition starts at $999. Replace the two hard drives with two SSD ($200) and add 16GB RAM ($100). Now show me a Supermicro that will have two nodes in 1RU for equivalent price. Then remind me how much it is going to sell in a year when I decide to upgrade. I guarantee that the Mac Minis would sell in a day on eBay.
    - theatrus2 14y ago
      You're not going to beat the two-nodes price. If you're fine with a single-NIC single-disk server, Minis are not a bad deal. However, you're not going to get dual-NICs, ECC memory, or multiple "spindles" in a Mini formfactor.

gonzo 14y ago

it was when he started talking about using solder on a 220 VAC connection that I lost the faith in him knowing how to do it right.

wlesieutre 14y ago
Out of curiosity, what's the problem with that? I assume he'd have the copper twisted together to make a good connection, with the solder just holding it in place instead of acting as the conductor.
- zalzane 14y ago
  depending on the resistivity of the copper wire versus the lead-tin mix in the solder, there may end up being more current going through the solder than the actual wire. Even if the resistivity values of the copper and solder are similar (I'm pretty sure they're at least in the same order of magnitude), you'll still end up with the same amount of current running through the copper wire and the solder.
  - Tloewald 14y ago
    Um, what? You'll dissipate more heat at 110v than 220v (high school physics -- heat dissipation is proportional to resistance and the square of current, so for a given wattage, it is inversely proportional to voltage — hence high voltage power transmission). In any event, 220v is no big deal. It's household current almost everywhere except the US. (A few countries use 110V, Australia is supposedly 240v.)
    - zalzane 14y ago
      The heat dissipated of the two metals would be identical if their resistances were the same, of course. I'm not even considering the difference in amperage between a 110V and 220V connection, as I'm pretty sure that's not what the parent parent post was in reference to. I was under the impression that the issue at hand was that using solder for high-load wires was bad because of solder's extremely low melting point. I meant to point out that even though the solder is just holding the wires together, there is still a current running through it, and if the resistivity of the solder is lower than that of the copper, the chances of the solder melting would be quite a bit higher. I honestly don't know if that's why the parent parent post mentioned that soldering high load wires are bad. I just extrapolated issues that someone might have when using solder on high load wires, and relayed them in my reply.

tzaman 14y ago

I'm a proud owner of a Mini Server (slightly customised - replaced memory and primary disk with SSD) for over a year. I use it as my main workstation and I love it; So small (and relatively cheap including the upgrade), yet so powerful.

jurre 14y ago

Before the thread becomes cluttered with people suggesting alternatives or questioning why you wouldn't just run <insert manufacturer, OS, etc>, the person that did this replied the following: simbimbo says: December 9, 2012 at 11:03 am Thanks for the great write up Hack A Day. I would like to answer some of the questions posted. @Geebles these machines all run SSD’s and I ordered them with AppleCare, so I hope to never have to change a drive ;-) As for the reason I built this.. Well, I guess I’m just like a challenge ;-), but seriously, the company I work for has a need to have large numbers of machines to build and test the software we make. There were plenty of discussions of Virtual environments and other “Bare Motherboard”/Google Datacenter-type solutions, but the fact is, the Apple EULA requires that Mac OS X run on Apple Hardware, since we are a software company we adhere to these rules without exception. These Mac Machines all run OS X in a NetBooted environment. We require Mac OS X because the products we make support Windows, Linux and Mac so we have data centers with thousands of machines configured with all 3 OS’s running constant build and test operations 24 hours a day 365 days a year. As for device failure, we treat these machines like pixels in a very large display, if a few fail, it’s ok, the management software disables them until we can switch them out. This approach allows us to continue our operations regardless of machine failures. @bitbass I tried the vertical approach, but manufacturing the required plenum to keep the air clean to the rear machines cost too much for this project, but it’s not off the table for the next rack @Kris Lee When I open the door I can literally watch the machine temps go up, but I can keep it open for 15-20 minutes before the core temps reach 180F @Adam Ahhh.. Nope, you can’t have my job ;-)

shardling 14y ago
I believe Mozilla has large racks of mac Minis for testing Firefox on. (Same issue, I believe -- to support multiple platforms on the same hardware, you need something that's legal to run OSX on.) e: Ah, according to [this article](http://www.wired.com/wiredenterprise/2012/05/mozillas-new-datacenter/ http://www.wired.com/wiredenterprise/2012/05/mozillas-new-da...) they have 500 Minis in their data center.
- deleted 14y ago
  [deleted]

frozenport 14y ago

Why not use 2 racks? This would solve his thermal dissipation problem and probably be easier when compared to getting custom hardware.

liquidise 14y ago
i would guess floor real estate constraints. Why do in 2 racks what you can do cost effectively in 1?
- keypusher 14y ago
  Because it's going to be extremely hot.
simbimbo 14y ago
Rack "foorprints" at the datacenter are expensive, and you pay for 15kw per footprint whether you use it or not. it just made sense to fit it all into one. Having had this rack running at full power for a couple of weeks now, I can say the temps stay lower than our SuperMicro racks in the same row!! and the Supermicro racks can only hold 20 machines before they run out of power in the footprint.

meaty 14y ago

I'm actually surprised any DC would take that equipment. They, in my experience at least, are very fussy about what what you put in the racks and power draw etc. Oh and we get 640 cores in 20U (8x4 core xeon machines each 1u) and that leaves enough room for a 32Tb SAN, FC switches and a pair of redundant LAN switches. REgarding splitting the power using the hack described, 160 melted minis and a halon cloud coming up. Looks pretty though.

jws 14y ago
REgarding splitting the power using the hack described, 160 melted minis and a halon cloud coming up. You should have a talk with your power cord provider. You should be using cables that can handle at least 4 amps in anything with a 110v plug on it. I don't think you can buy one smaller than 18ga and those are good for 10 amps. Remember, you have to handle enough current to blow the breaker if something goes wrong (unless you are British and have your own fuse in the plug).
- simbimbo 14y ago
  We went through several designs with the vendor. This cable is designed to operate at the circuit rating. The wire from the PDU plug to the split is 12GA 600V and the cable from the split to the mini is 14GA 300V.
michaelt 14y ago
A 1U xserve maxes out at 334 Watts [1]. Each Mac mini takes 85 Watts [2], so 4 is 340 Watts. At 220v that's 1.54 Amps. Is splitting the power really so dangerous? [1] http://support.apple.com/kb/HT3513 http://support.apple.com/kb/HT3513 [2] http://support.apple.com/kb/HT3468 http://support.apple.com/kb/HT3468
alexkus 14y ago
It must be their own DC. Most rackspace rented (in UK DC's at least) tend to be a maximum of 16A (at 240V) per 42U cabinet, so just under 4kW. By my estimation those Mac Minis will be drawing ~13kW at peak.

w3pm 14y ago

I understand the idea of treating them like pixels, so if a fan dies or a NIC card dies, no problem, just stop using that Mini. But what about memory corruption or other issues that are more difficult to detect? Normally server hardware has things like ECC memory to prevent these issues, but in this case a Mini with bad RAM could intermittently corrupt data for some time before it's noticed (if ever).

lallysingh 14y ago
The machines are for testing. They'll detect those through secondary means. If a machine's faulty, it'll cause two cases: (1) faulty software will register as faulty; (2) good software will register as faulty. The third case (faulty software marked as good), is really unlikely, and any time it does happen, a later bug report will give a hint. A test failure will probably bring up an engineer that will track down the issue, and a re-test will inevitably occur. The faulty machine will eventually (hopefully) get labeled flaky and will get repaired. Of course, nobody may care and just use a double-test to verify that an executable is good.
- rdtsc 14y ago
  > A test failure will probably bring up an engineer that will track down the issue, and a re-test will inevitably occur. The faulty machine will eventually (hopefully) get labeled flaky and will get repaired. Depending on how valuable the engineers' time is. I have seen this played out like this: hardware gets blamed last after hours and days of testing have been wasted. So tests are run and re-run, blame goes all around until finally after hours and hours of testing it is determined that maybe it is hardware after all. In the end an engineers' time is worth a lot more than savings obtained by running flaky but cheaper hardware.

rdtsc 14y ago

Mac Minis are horrible server hardware. We've had a couple running as servers. They fail randomly. Their hard drives fail. They don't rack mount easily. The only reason to have them is if you inherit some old ones, don't want to throw them away, and then don't mind replacing and throwing failed units away pretty often.

sliverstorm 14y ago
Well, they do provide good machine density when you rack them like he did.
- rdtsc 14y ago
  Yap definitely true.
taligent 14y ago
All servers fail randomly and so do hard drives so not sure why Mac Mini would be any different. Also they rack mount very easily: http://www.sonnettech.com/product/rackmacmini.html http://www.sonnettech.com/product/rackmacmini.html
- rdtsc 14y ago
  They do but we found that mac minis do so more often. I have a sample of about 15 out of say 100+ other regular 1U rack mountable machines. Mac minis failed at a much higher rate. The worst part is that some failures were not a 'stop dead failure' where it was easy to detect and replace. Some would just freeze intermittently, some just got really slow. One could tell that there is a practical difference between server grade and consumer grade components. In retrospect time spent debugging and messing with this was probably not worth the gain in density and the cost.
  - adrianpike 14y ago
    I for one would love it if you'd have the chance to dig a little bit and share what you find. I've only got a sample size of two recently purchased (2012) Mini's, and they've both had memory issues.

datums 14y ago

Definitely a fun challenge. If you're going to invest in the hardware and custom build. Forget the y cable. Figure out a better solution. Rent 1/2 rack next to it to hold the pdus. +1 on the massive door fans.

patangay 14y ago

We did something similar at Facebook for iOS and OSX automated testing and a few of them doing iOS app builds. Here is a post that Jay Parikh (VP of Infrastructure) made about it. http://tinyurl.com/cnvss4v http://tinyurl.com/cnvss4v Our density isn't as high (we have 64 minis) because of cooling and cabling that we designed according to our datacenter cooling standards. @jurre - If you want to chat about our design, message me and I can put you in touch with our hardware designer.

simbimbo 14y ago
I would like to chat about it.
i386 14y ago
Would love to hear more about your automated testing!
jurre 14y ago
I just copied that reply from the person that built the actual rack, but I would love to hear more about your iOS testing infrastructure/process!
- jevinskie 14y ago
  Me too! Before we used fruitstrap[0] we used DeviceAnywhere. This was a 2U rack unit that contained a franken-iPhone. The hardware buttons were soldered up to GPIO on the DeviceAnywhere. [0]: https://github.com/ghughes/fruitstrap https://github.com/ghughes/fruitstrap
patangay 14y ago
For those who showed interest in learning more about our infrastructure/automated testing process, if you could drop me an email? gp at our corp domain fb.com. I don't work on the team anymore, but I can probably start off a thread with the right people involved from Facebook's side.

j45 14y ago

I would love to get the specs on the tray that he put the 4 mac mini's on.

simbimbo 14y ago
They're in production now.. H-Squared. great vendor. http://h-sq.com/products/minirack/index.html http://h-sq.com/products/minirack/index.html
- j45 14y ago
  Great, thanks for the pointer!

georgebarnett 14y ago

Interestingly, it looks like the front fans blow _into_ the rack. This means that if the door isn't securely closed it'll blow open - being on hinges and having massive fans attached. It would be better to have the fans on the back and suck air through the rack rather. That said, DC floor space is cheap compared to power and cooling. I'm surprised they didn't lower the density so as not to have a massive fire risk.

simbimbo 14y ago
I looked into some rack cooling options, but was unable to find a solution that would provide the amount and wide coverage of airflow I needed to move air slowly and uniformly through the rack to provide effective cooling. ( I was designing this to be used with the active cooling rear doors, so I couldn't overwhelm the door with too much air or it wouldn't cool the air effectively, and would raise the ambient temp of the room). So the fans move a high volume of air through the entire cabinet (including the corners) at low velocity resulting in very effective cooling of each of the 40 shelves. The fans are large so they move a high volume of air at a low speed, the door doesn't move is left unlatched. Also, can you please explain your "Massive Fire risk" comment? All of the hardware installed in this rack is UL certified and all of the machines will simply shut down if they get too hot.

lallysingh 14y ago

I'm surprised both that the high density worked for consumer devices, and that the rack wasn't prettier.

mewmoo 14y ago

Yes.. but why?

nsxwolf 14y ago

Curious - what do we call a computer like this? It's obviously not going to make the TOP500, but is it a "supercomputer"? I thought perhaps "minisupercomputer" might be fitting, but according to Wikipedia that is a term for a class of computers that became obsolete in the early 90s.

sproketboy 14y ago

Yeah but can it run Crysis? /s