4 ms·

So I'm the guy who rack-mounted a bunch of game machines for https://kentonshouse.com https://kentonshouse.com -- and am currently nearing completion of a new h

So I'm the guy who rack-mounted a bunch of game machines for https://kentonshouse.com https://kentonshouse.com -- and am currently nearing completion of a new house with even more machines. Here's some of the challenges I've personally encountered in this. *Fitting the GPU* A normal GPU, mounted in the normal way (perpendicular to the motherboard), will not fit in a 2U case. It looks like Jeff's solution was to buy a half-height GPU. Neat! I didn't actually realize those exist. But it looks like NVidia's regular consumer-oriented GPUs don't come in this form factor; Jeff instead went with an RTX A2000, which is part of NVidia's "professional" GPU line. The problem with this is that it's way more expensive to achieve the same gaming performance. In my previous house I solved the problem by using 3U chassis. This provides just enough space to mount the GPU perpendicular. However, it still had problems. Modern GPUs require supplementary power direct from the PSU. The socket for this usually points outward, away from the motherboard. In a 3U chassis, this socket is right up against the top plate so there is no room to plug in the power. Sometimes you can find a GPU that has the power socket pointing in a different direction, but they have become rare. I actually had to physically mod ten GTX 1060's by hand, cutting away the plastic socket housing and directly attaching each power wire to the corresponding pin. Big PITA but it worked. In the new house I'm going with 2U, but using a riser card. So, the GPU is mounted parallel to the motherboard. This seems to work OK. The big problem is ensuring the chassis and the motherboard both have their expansion slots aligned right for this. Most 2U chassis are designed for perpendicular, half-height expansion cards. A smaller fraction support full-height parallel cards on a riser. The motherboard also has to have a PCIe slot in the correct place for the riser. This is usually the "top" slot, closest to the CPU, and I've noticed a lot of motherboards these days are opting to omit that slot to make more space around the CPU. The chassis I'll probably go with is the iStarUSA D-200: https://www.istarusa.com/en/istarusa/products.php?model=D-200 https://www.istarusa.com/en/istarusa/products.php?model=D-20... (I'm not too excited about it though, and would be happy for other recommendations.) At least one motherboard that worked when I tried it was the ASUS PRIME B660-PLUS, but I haven't made a final motherborad selection yet. A further problem which might be specific to this chasis is that the total horizontal space between the riser card and the power supply is tight. Some GPUs are excessively "tall" and won't fit. (The same problem probably applies to 3U chassis with perpendicular mounting.) I was just barely able to fit the EVGA GeForce RTX 3070 Ti 08G-P5-3785-K into the space. *RAM orientation* Servers orient their RAM modules parallel to the direction of air current, that is, perpendicular to the back of the chasis. For reasons I don't understand, consumer / gaming motherboards almost never do this, they orient the RAM parallel to the back. Maybe this is because desktop chassis are not usually optimized for efficient airflow anyway so it doesn't matter? And there's more space on the board if the RAM can be mounted this way? I dunno. Anyway, in my experiments so far this has just not caused a problem. Yeah, the RAM has non-ideal orientation, but there haven't been any issues with overheating. ::shrug:: *PSU* Servers use a different form factor of PSU. This is infuriating because all the server PSUs available retail seem to be terrible compared to desktop PSUs! They tend not to support higher-end wattage (needed for power-hungry GPUs) and they are never "modular" (you get a bundle of cords sticking out of the PSU and there's no way to remove the ones you don't need except with scissors). There are some 2U chassis that actually allow you to mount a desktop PSU. However, in practice they are not actually compatible with any modern desktop PSU, because all such PSUs these days are designed with their intake fan on the top or bottom, not on the front/back. In a 2U chassis, that intake will be totally blocked by the top/bottom of the chassis. I am going to have to live with crappy server PSUs I guess. *Retail server market is awkward* More generally, I get the general feeling that the server-oriented components which are available via normal retail channels like NewEgg are... not what big server farms actually use. Everything seems kinda crappy? Maybe I just feel this way because they don't candy-coat everything the way the consumer market does. But I have the funny feeling that people building out datacenters generally order mass quantities of chassis, PSUs, and even motherboards tailored specifically to their needs, and as a result the retail market for server components may be surprisingly small and poorly served. But that's just my suspicion based on my experiences here.

justinlloyd 4y ago
SuperMicro do sell modular power supplies and you can fit their workstation power supplies to some of their server chassis. Depending on the SuperMicro chassis, the power supply may not even have any cables at all, it simply slots in, and is hot swappable, and it is the chassis that has the cabling, and again, some of those are modular. You can also make a modular power supply out of a regular supply, though it is a bit of work obviously. One of my SuperMicro (CSE-846) cases has 1280w PSUs and the other SuperMicro has a 2kW PSU, both are 80+ Platinum. Not all server chassis obviously, but many have fans mounted at the front and pull air through the case, over the motherboard, through some plastic ducting, and out the back. Some server designs have only a heat sink on the CPU, and rely on this ducted air flow from the case fans to keep everything cool. RAM header orientation will often be due to CPU, VRM and memory controller orientation, but mostly it is about real-estate on the motherboard and physics. Retail market for server chassis is pretty abysmal. You might consider a Cooler Master, or a SuperMicro or an ASUS case. Depends on your budget and your willingness to use a Dremel. If you are considering a 2U rack mount case with a full-sized GPU, you might want to look at cases that specifically mount the GPU sideways, there are quite a few out there that will happily take two GPUs, though the cases are often full-depth. Just be aware that server GPUs and consumer grade GPUs have different cooling options and consumer grade cards may not fair well if the case is too close to the exhaust fan. Instead of a riser, and fighting for a PCIe slot, you might also consider a PCIe extension cable, and then fabricate a custom bracket to hold the GPU in the desired orientation.
error503 4y ago
There's just not much serious market for a 'build your own' server platform. Almost anyone serious buys systems prebuilt from Dell/HP/Lenovo, which have nice integrated solutions solving all of these problems, but the elements are too tightly coupled to work well in a piecemeal environment. For example, power might be distributed entirely using a backplane that the PSUs, fan trays, and motherboard all plug into. Since the market for DIY is so small, there's no real push for standardization, which would support a modular environment like standard PCs, and there's some disincentive too, since the vendors want flexibility to do cooling etc. in the best way. So what you end up with is a combination of low-volume bespoke solutions and 'ugly hacks' to make the high-volume commodity parts work in a way they weren't really designed to. The closest that exists out there is probably SuperMicro. They have a decent variety of chassis, backplanes, motherboards and so on that work well together in a system, and the stuff is pretty decent quality. But because they are still designed for 'serious' workloads, they will not hit the cost of an equivalent desktop PC.

justinlloyd 4y ago

SuperMicro do sell modular power supplies and you can fit their workstation power supplies to some of their server chassis. Depending on the SuperMicro chassis, the power supply may not even have any cables at all, it simply slots in, and is hot swappable, and it is the chassis that has the cabling, and again, some of those are modular. You can also make a modular power supply out of a regular supply, though it is a bit of work obviously. One of my SuperMicro (CSE-846) cases has 1280w PSUs and the other SuperMicro has a 2kW PSU, both are 80+ Platinum. Not all server chassis obviously, but many have fans mounted at the front and pull air through the case, over the motherboard, through some plastic ducting, and out the back. Some server designs have only a heat sink on the CPU, and rely on this ducted air flow from the case fans to keep everything cool. RAM header orientation will often be due to CPU, VRM and memory controller orientation, but mostly it is about real-estate on the motherboard and physics. Retail market for server chassis is pretty abysmal. You might consider a Cooler Master, or a SuperMicro or an ASUS case. Depends on your budget and your willingness to use a Dremel. If you are considering a 2U rack mount case with a full-sized GPU, you might want to look at cases that specifically mount the GPU sideways, there are quite a few out there that will happily take two GPUs, though the cases are often full-depth. Just be aware that server GPUs and consumer grade GPUs have different cooling options and consumer grade cards may not fair well if the case is too close to the exhaust fan. Instead of a riser, and fighting for a PCIe slot, you might also consider a PCIe extension cable, and then fabricate a custom bracket to hold the GPU in the desired orientation.

error503 4y ago

There's just not much serious market for a 'build your own' server platform. Almost anyone serious buys systems prebuilt from Dell/HP/Lenovo, which have nice integrated solutions solving all of these problems, but the elements are too tightly coupled to work well in a piecemeal environment. For example, power might be distributed entirely using a backplane that the PSUs, fan trays, and motherboard all plug into. Since the market for DIY is so small, there's no real push for standardization, which would support a modular environment like standard PCs, and there's some disincentive too, since the vendors want flexibility to do cooling etc. in the best way. So what you end up with is a combination of low-volume bespoke solutions and 'ugly hacks' to make the high-volume commodity parts work in a way they weren't really designed to. The closest that exists out there is probably SuperMicro. They have a decent variety of chassis, backplanes, motherboards and so on that work well together in a system, and the stuff is pretty decent quality. But because they are still designed for 'serious' workloads, they will not hit the cost of an equivalent desktop PC.