3 ms·
In the past I've built servers by carefully selecting parts, calculating maximum power usage versus airflow, calculating full time electrical load, building, te
by johnklos 4y ago
In the past I've built servers by carefully selecting parts, calculating maximum power usage versus airflow, calculating full time electrical load, building, testing with fans disabled in rooms with high ambient temperatures, looking for failure modes, researching failure rates of specific power supplies and drives, et cetera.
I'd build one or two, test them for several months while making adjustments and fixing various weaknesses, then I'd build ten, twenty, however many were needed.
One particular client decided they wanted to skip all that and just spend the money on Dell. They didn't have many servers, certainly not enough to justify a separate server room. Their offices weren't air conditioned at night, and they had an entire summer where one server or another would become unresponsive over the course of a weekend, and sometimes at night between weekdays. Accessing iDRAC was beyond what they wanted to do, so of course I had to do that.
They had Dell support, but Dell had no "fix" for unstable machines other than to tell them to build a server room. Mind you - the ambient temperatures were always below 100º - any reasonable person would say that while that's not ideal for servers, "premium" servers should still be able to handle warm rooms, particularly when they're idle, and not crash or lock up.
After that fiasco, they gradually replaced the Dells, one at a time, with machines I built. I wish they had tried harder to return the Dells, but they just wrote them off.
I've learned that any savings in time and money (mostly - the Dells cost more than the machines I build, even accounting for extra billable time) aren't worth the loss in time, productivity and reliability in the long run. Of course, the opposite would be true if I just wanted more billable work, but I can't do that, unlike many others in the field.
BTW - when I build a new generation of servers, I do months of testing, but the same general platform can last a good five years, like Ryzen 1000 through 5000 systems with ECC have lasted since 2017.
- flyinglizard 4y agoI might have a consulting gig for you if you’re interested. Let me know how to get in touch if relevant!
- johnklos 4y agoSure, I'd be interested. My email address is john at klos dot com :)
- c_o_n_v_e_x 4y agoIs there much business building custom rack mount servers for people? I would have thought that market would have been taken out by the big brands. Do you have a favored brand for server motherboards?
- nehal3m 4y agoI do something similar as part of a team that builds bespoke equipment (can't be too specific). It's a full time job for 2 people for runs of around 50 custom half-racks every year. It's definitely a niche, but there's demand for sure.
- johnklos 4y agoASRock makes some good motherboards, and all support ECC by default. Somewhat related: I don't like server motherboards like those from, say, Supermicro, because they don't care very much about security. For example, they've had motherboards for years that let you share the primary ethernet with IPMI. Is there a way to disable this? No. Is there a way to force IPMI to ONLY use the dedicated ethernet? No - the settings are stored in the battery backed memory. Wouldn't a jumper make sense? Sure, but they don't think so. What this means is that if your motherboard battery dies and your dedicated IPMI port is somehow disconnected, your server can be completely, 100% taken over via the public interface. While it's not a likely attack scenario, the point is that if you have a need to ship servers to various datacenters, you genuinely can't be sure that your server won't be utterly vulnerable on first boot unless you're there, confirming it yourself. Does Supermicro care? No. They refuse to consider the loss of all security due to a dead battery to be a security issue. Having a security issue is one thing - ignoring it when told about it (or worse - pretending it's not an issue) is something else entirely.