6 ms·

> I'm no expert on the server market though. Could someone with a bit more knowledge chime in with a how big a threat this really is? Sys Admin here. People ar

> I'm no expert on the server market though. Could someone with a bit more knowledge chime in with a how big a threat this really is? Sys Admin here. People are starting to notice and raise eyebrows. It is certainly an option, especially for new stacks. The power/price/performance ratios on AMD are truly amazing. It certainly helps that Amazon, Microsoft, Cray, and Baidu have all already started to use Epyc servers.

mroche 8y ago
Student admin myself here at Uni. With Dell being our primary provider I did some general comparisons with the currently available EPYCs. Our Dual socket Skylakes (36 total cores without hyperthreading) with 128GB of ECC ram cost ~18K each. With an EPYC chip I could build 48+ core systems with 256GB RAM for ~3K less. If these next gens have similar pricing, they look very competitive for the future of on-premise (and cloud) HPC stacks.
- sitkack 8y ago
  I was in your same place exactly 20 years ago when I convinced the dept head that we (mostly me) could build our own machines for the test lab instead of purchasing units from corp it. I should have productionized the building but otherwise it went pretty well. 1, you are right, you can build your own machines for way less than dell 2, you are now the support team. So when you get a call months, years later about hardware quirks, missing pieces or odd incompatibilities, expect it. 3, Buy extras and keep the machines largely the same. The fleet will be its own spare parts as it ages out. You will need extras for differential testing. 4, Keep a log and copy of all the bioses supplied by the motherboard manufacturer. Baseline and benchmark each server before it is put into production, retest as it is cycled out for other duties. 5, If you don't max out the memory when built, do track ram prices and there is a sweet spot of a dip just as the ram tech ages out. If you miss it, ram prices go back up again. An older machine with lots of ram will useful for a very long time. If one plans it right, 100 node HPC cluster decays into a 75 node render farm into a 25 node distributed system test cluster, I think they could get 10 years of service out of the purchase.
  - mroche 8y ago
    I don’t disagree with any of that. I usually prefer building my own when I’m the maintainer/builder. But not my current situation, a few things to note: * I have (hopefully) one semester left. After that, I’m gone. * A good portion of our experienced team will be gone within the next two years, including the few grad students who helped build our cluster. Introducing “non-supported” hardware would likely introduce problems for the next group. * The prices above were from Dell, not self assembly shopping. University policies mandate any university funded tech/infrastructure equipment (especially desktops/servers) be done through specific partners. For us, that’s Dell, GovConnection, and CDW. I host my own 6 node Supermicro Microblade setup (render farm) from CDW that took almost a full year to purchase because nobody in the Purchasing Dept. for clubs was aware of that rule. Until my hopping from Boxx to Thinkmate to “haha, funny story...” uncovered it.
    - sitkack 8y ago
      > prices above were from Dell, not self assembly shopping That is amazing. With those kind of prices folks might upgrade before their old hardware has properly aged out. At what point does AMD institute a buyback program for Intel hardware? (joking)

mroche 8y ago

Student admin myself here at Uni. With Dell being our primary provider I did some general comparisons with the currently available EPYCs. Our Dual socket Skylakes (36 total cores without hyperthreading) with 128GB of ECC ram cost ~18K each. With an EPYC chip I could build 48+ core systems with 256GB RAM for ~3K less. If these next gens have similar pricing, they look very competitive for the future of on-premise (and cloud) HPC stacks.

sitkack 8y ago
I was in your same place exactly 20 years ago when I convinced the dept head that we (mostly me) could build our own machines for the test lab instead of purchasing units from corp it. I should have productionized the building but otherwise it went pretty well. 1, you are right, you can build your own machines for way less than dell 2, you are now the support team. So when you get a call months, years later about hardware quirks, missing pieces or odd incompatibilities, expect it. 3, Buy extras and keep the machines largely the same. The fleet will be its own spare parts as it ages out. You will need extras for differential testing. 4, Keep a log and copy of all the bioses supplied by the motherboard manufacturer. Baseline and benchmark each server before it is put into production, retest as it is cycled out for other duties. 5, If you don't max out the memory when built, do track ram prices and there is a sweet spot of a dip just as the ram tech ages out. If you miss it, ram prices go back up again. An older machine with lots of ram will useful for a very long time. If one plans it right, 100 node HPC cluster decays into a 75 node render farm into a 25 node distributed system test cluster, I think they could get 10 years of service out of the purchase.
- mroche 8y ago
  I don’t disagree with any of that. I usually prefer building my own when I’m the maintainer/builder. But not my current situation, a few things to note: * I have (hopefully) one semester left. After that, I’m gone. * A good portion of our experienced team will be gone within the next two years, including the few grad students who helped build our cluster. Introducing “non-supported” hardware would likely introduce problems for the next group. * The prices above were from Dell, not self assembly shopping. University policies mandate any university funded tech/infrastructure equipment (especially desktops/servers) be done through specific partners. For us, that’s Dell, GovConnection, and CDW. I host my own 6 node Supermicro Microblade setup (render farm) from CDW that took almost a full year to purchase because nobody in the Purchasing Dept. for clubs was aware of that rule. Until my hopping from Boxx to Thinkmate to “haha, funny story...” uncovered it.
  - sitkack 8y ago
    > prices above were from Dell, not self assembly shopping That is amazing. With those kind of prices folks might upgrade before their old hardware has properly aged out. At what point does AMD institute a buyback program for Intel hardware? (joking)