2 ms·

I wanted remote management (IPMI), which none of the Threadripper boards offered. I went with the ASRock Rack ROMED8-2T, which also has 2x 10G Ethernet on board

I wanted remote management (IPMI), which none of the Threadripper boards offered. I went with the ASRock Rack ROMED8-2T, which also has 2x 10G Ethernet on board, which was another nice thing I didn't have to sacrifice a PCIe slot for. It does require a Tower Case with space for fans on top though, because the CPU slot is rotated 90 Degrees compared to Threadripper boards, so the airflow is different. The EPYC CPU was also a quite a bit cheaper than the then-equivalent Threadripper 2950X (though the mainboard being $600 made up for that). This is even more true today because AMD really jacked up the prices for Threadripper to the point that EPYC is actually a good budget alternative. I guess that making 16 Core Ryzen made low-end Threadrippers less attractive, but it's the PCIe slots that were so great about those! Also, I do believe that it was much easier to find 64 GB RDIMMs whereas 64 GB ECC UDIMMs were not available or much more expensive, though my memory (ha!) is hazy on that, I just remember it being a PITA. So that EPYC system was just much more compelling.

paulmd 3y ago
ROMED8-2T is one of the all-star boards of the modern era imo. Like that's literally "ATX-maxxed" in the Panamax sense - you can't go bigger than that in a traditional ATX layout, and there is no point to having a bigger CPU (even if you do not use all the pins) because it starts to eat up the space for the Other Stuff. It's a local optimum in board design and everything else past here gets weirder and has to start making more and more tradeoffs. EEB/EE-ATX can push things a little farther (like GENOAD8X-2T) but you can't pull any more PCIe slots off, so it has to be MCIO/oculink instead. And imo this is the reasonable limit of what can be done with single-socket Epyc even in EEB, this is "EEB-max". And you can't really get more than 8 memory slots without moving the CPU over to the other side of the board, like MZ32-AR0 or MZ33-AR0, which means it overhangs the PCIe slots etc. IIRC you can sorta do 16-dimm SP3 if you don't do OCP 2.0 (gigabyte or asus might have some of these iirc) and you drop to like 5 pcie slots or something. But it's really hard to get 2DPC on epyc at all, the layouts get very janky very quickly. You can fit more RAM slots into EEB/EE-ATX with a smaller socket (dual 2011-3 with 3DPC goes up to 24 slots in EE-ATX) but 2DPC is as big as you can go with epyc in a commodity form-factor. In SP5 this gets fully silly, MZ33-AR0 is an example of 2DPC 12-channel SP5, and it's like, oops all memory slots, even with EEB and completely overlapping every single pcie slot. And of course dual-socket epyc gets very cramped even on EEB/EE-ATX even with only 8 slots per socket (MZ72-HB0). You just are throwing away a tremendous amount of board space and you lose pcie, MCIO, everything. SP3 is already a honkin big socket let alone SP5, let alone two SP3, let alone two SP5, etc... they are big enough that you have to make Tough Choices about what parts of the platform you are going to exploit, or accept a non-"standard" form factor (it's not standard for anyone except home users/beige boxes). Servers don't use EEB/EE-ATX form factors anymore, because it just isn't the right shape for these platforms. And you need to be pulling a significant amount of the IO off in high-density formfactors (MCIO, Oculink, SlimSAS, ...) already, and your case ecosystem needs to support that riser-based paradigm, etc. ATX is dying and enthusiasts are not even close to being ready for the ground to shift underneath them like this. (me when I see a honkin' pair: https://www.gigabyte.com/us/Enterprise/Server-Motherboard/MZ73-LM1-rev-1x https://www.gigabyte.com/us/Enterprise/Server-Motherboard/MZ...) There's still good AM4, AM5, and LGA1700 server boards (with ECC) btw - check out AM5D5ID-2T, X570D4I-2T, X470D4U, W680 ACE IPMI, W680D4U-2L2T/G5, X11SAE-M, X11SAE-F, IMB-X1231, IMB-X1314, X300TM-ITX, etc. And Asrock Rack and Supermicro do make threadripper boards too (WRX80D8 family, TRX40D8 family, etc), although I think they're not viable since threadripper is leaning farther and farther into the OEM market and it just doesn't make cost sense unless you really need the clocks. It's not like the X99 days where HEDT was just "better platform for enthusiasts", there is a big penalty to choosing HEDT right now if you don't need it, and it's generally too segmented to make sense. Unregistered DDR4 tops out at 32GB per stick (UDIMM or SODIMM), registered can go larger. DDR5 unregistered will go larger, and actually a few 48GB sticks do exist already, but generally you can't use all four slots without a massive hit to clocks (current LGA1700/AM5 drop to 3600 MT/s) so consumers/prosumers have to consider that one carefully. (this generally means that drop-in upgrades are not viable for DDR5 memory btw - 4-stick configs suck, you should plan on just buying 2 new sticks when you need more. And the slots on the mobo are worse than useless, since the empty slots worsen the signal integrity compared to 2-slot configurations without the extra parasitics...)
- MenhirMike 3y ago
  I agree, the ROMED8-2T has everything I want and compromises almost nothing. One of the PCI Express slots is shared with one of the on-board M.2 slots, SATA, and Oculink, but even then, you get to choose: Run the slot at x16 and turn off M2/Sata/Oculink? Run the Slot in x8 and get M2/Sata but lose Oculink? Or disable the slot and get M2/Sata/Oculink? I think that's a great compromise (I run the slot at x8 and use it for a Fibre Channel card to my backup tape drive). Lovely block diagram in the manual as well. Plenty of Fan headers as well, and using SFF-8643 connectors for the SATA ports makes so much sense (though it's an extra cost for the cables). They even put a power header if you run too many high-powered PCIe cards (since PCIe allows AFAIK to pull up to 75W from the slot). They really put every feature that makes sense onto that board, and yeah, if you want Dual CPUs or 16 DIMM Slots, chances are that a proper vendor server is more what you want. I can't think of anything that I don't like about the board. Well, I wish the built-in Ethernet ports weren't RJ45 but SFP+, but that's really the only thing I wish to change.
  - paulmd 3y ago
    > I can't think of anything that I don't like about the board. Well, I wish the built-in Ethernet ports weren't RJ45 but SFP+, but that's really the only thing I wish to change. YES. Jesus. The fact that the state of the art for SFP+ motherboards is basically still Denverton (C3758 etc) is embarrassing. You have all these "server" motherboards with consumer base-t standards... even if you have base-t to the workstations, surely your fancy SOHO/SMB setup will have a server closet where it would make sense to have SFP+ for the server, to punch down a bunch of individual base-T links... (and actually base-t has much more latency than SFP+ as well - measurable on NVMe drives etc) In theory this is something that OCP 2.0 mezzanine cards (like MZ31-AR0 uses) can do for you. Actually these are quite cheap because of the very limited surplus market for them, and you can get adapter cards to put them in PCIe slots if you want (the adapter cards are unusual enough they're not cheap, but like most pcie adapter cards they're not inherently expensive or electrically difficult). So you can put SFP+ on anything you want with a OCP 2.0 slot - but of course most of the asrock rack, supermicro, etc are all base-t with no OCP 2.0. Infuriating. (OCP 2.0 does make "traditional" IO area very difficult however, it eats a lot of space in that IO shield area, and this often has the side-effect that the CPU gets pushed over into the other side of the board where it starts overlapping pcie slots etc. There are reasons to not do it - and this is another problem brought on by the ATX layout. But then offer some SFP+ please - really SFP28 25gbit should be available at this point on things like the ROMED8 model imo.) And before someone says Minisforum MS-01... no ECC (which could be forgiven) and the chinese vendors are unfortunately quite poor in the support department in general. They are great at marketing via influencers etc but I have read a number of people say they've run them long-term and were upset about the support story. Which is a shame because on paper it's quite attractive - 12/13th gen are incredibly speedy (faster than AMD 7000 series) and most server workloads don't benefit that much from v-cache, a laptop CPU (again, ideally with ECC) with a couple SFP28 cards is more or less an ideal 25gb switch, can serve NVMe flash drives pretty fast, etc. It is quite desirable to have high per-thread performance in a homelab, especially when you are the sole user (4 people running 1 GB/s is not the same as 1 person running 4 GB/s). i3 7100 was actually an extremely interesting sku for this reason - 3.9 GHz dual-core (7350K is 4.2 GHz base, and 8350K/9350K were 4.2 GHz base 4C4T) is quite punchy, you never drop clocks due to AVX offset, and i3s supported ECC in this era. AM4 server boards weren't mature yet (X470D4U was one of the first good ones and it still took several years to stabilize fully) so the alternative was sandy bridge xeons etc, and the 7100 (while not a popular gaming CPU) absolutely destroyed that shit for homelab NAS builds, in both perf and efficiency. Again, it's kind of a tragedy that C3758 is the norm still - that's 8x 14nm e-cores with RDIMM support and onboard Intel QAT (of whatever gen). That is not fast at all, we are talking like sub-zen1 performance here probably, with no AVX, etc. (Unfortunately minisforum's AMD boards are not any better in the support department, and AMD segments ECC to the Pro laptop chips too, etc.) I think in practice the problem is the length of the SFP+ cage - it's noticeably longer than base-t. And that means either you're wasting space on your base-t boards, or you have to design a custom layout for SFP+, which is already a (truthfully) small/niche market etc. It is understandable, just unfortunate.

paulmd 3y ago

ROMED8-2T is one of the all-star boards of the modern era imo. Like that's literally "ATX-maxxed" in the Panamax sense - you can't go bigger than that in a traditional ATX layout, and there is no point to having a bigger CPU (even if you do not use all the pins) because it starts to eat up the space for the Other Stuff. It's a local optimum in board design and everything else past here gets weirder and has to start making more and more tradeoffs. EEB/EE-ATX can push things a little farther (like GENOAD8X-2T) but you can't pull any more PCIe slots off, so it has to be MCIO/oculink instead. And imo this is the reasonable limit of what can be done with single-socket Epyc even in EEB, this is "EEB-max". And you can't really get more than 8 memory slots without moving the CPU over to the other side of the board, like MZ32-AR0 or MZ33-AR0, which means it overhangs the PCIe slots etc. IIRC you can sorta do 16-dimm SP3 if you don't do OCP 2.0 (gigabyte or asus might have some of these iirc) and you drop to like 5 pcie slots or something. But it's really hard to get 2DPC on epyc at all, the layouts get very janky very quickly. You can fit more RAM slots into EEB/EE-ATX with a smaller socket (dual 2011-3 with 3DPC goes up to 24 slots in EE-ATX) but 2DPC is as big as you can go with epyc in a commodity form-factor. In SP5 this gets fully silly, MZ33-AR0 is an example of 2DPC 12-channel SP5, and it's like, oops all memory slots, even with EEB and completely overlapping every single pcie slot. And of course dual-socket epyc gets very cramped even on EEB/EE-ATX even with only 8 slots per socket (MZ72-HB0). You just are throwing away a tremendous amount of board space and you lose pcie, MCIO, everything. SP3 is already a honkin big socket let alone SP5, let alone two SP3, let alone two SP5, etc... they are big enough that you have to make Tough Choices about what parts of the platform you are going to exploit, or accept a non-"standard" form factor (it's not standard for anyone except home users/beige boxes). Servers don't use EEB/EE-ATX form factors anymore, because it just isn't the right shape for these platforms. And you need to be pulling a significant amount of the IO off in high-density formfactors (MCIO, Oculink, SlimSAS, ...) already, and your case ecosystem needs to support that riser-based paradigm, etc. ATX is dying and enthusiasts are not even close to being ready for the ground to shift underneath them like this. (me when I see a honkin' pair: https://www.gigabyte.com/us/Enterprise/Server-Motherboard/MZ73-LM1-rev-1x https://www.gigabyte.com/us/Enterprise/Server-Motherboard/MZ...) There's still good AM4, AM5, and LGA1700 server boards (with ECC) btw - check out AM5D5ID-2T, X570D4I-2T, X470D4U, W680 ACE IPMI, W680D4U-2L2T/G5, X11SAE-M, X11SAE-F, IMB-X1231, IMB-X1314, X300TM-ITX, etc. And Asrock Rack and Supermicro do make threadripper boards too (WRX80D8 family, TRX40D8 family, etc), although I think they're not viable since threadripper is leaning farther and farther into the OEM market and it just doesn't make cost sense unless you really need the clocks. It's not like the X99 days where HEDT was just "better platform for enthusiasts", there is a big penalty to choosing HEDT right now if you don't need it, and it's generally too segmented to make sense. Unregistered DDR4 tops out at 32GB per stick (UDIMM or SODIMM), registered can go larger. DDR5 unregistered will go larger, and actually a few 48GB sticks do exist already, but generally you can't use all four slots without a massive hit to clocks (current LGA1700/AM5 drop to 3600 MT/s) so consumers/prosumers have to consider that one carefully. (this generally means that drop-in upgrades are not viable for DDR5 memory btw - 4-stick configs suck, you should plan on just buying 2 new sticks when you need more. And the slots on the mobo are worse than useless, since the empty slots worsen the signal integrity compared to 2-slot configurations without the extra parasitics...)

MenhirMike 3y ago
I agree, the ROMED8-2T has everything I want and compromises almost nothing. One of the PCI Express slots is shared with one of the on-board M.2 slots, SATA, and Oculink, but even then, you get to choose: Run the slot at x16 and turn off M2/Sata/Oculink? Run the Slot in x8 and get M2/Sata but lose Oculink? Or disable the slot and get M2/Sata/Oculink? I think that's a great compromise (I run the slot at x8 and use it for a Fibre Channel card to my backup tape drive). Lovely block diagram in the manual as well. Plenty of Fan headers as well, and using SFF-8643 connectors for the SATA ports makes so much sense (though it's an extra cost for the cables). They even put a power header if you run too many high-powered PCIe cards (since PCIe allows AFAIK to pull up to 75W from the slot). They really put every feature that makes sense onto that board, and yeah, if you want Dual CPUs or 16 DIMM Slots, chances are that a proper vendor server is more what you want. I can't think of anything that I don't like about the board. Well, I wish the built-in Ethernet ports weren't RJ45 but SFP+, but that's really the only thing I wish to change.
- paulmd 3y ago
  > I can't think of anything that I don't like about the board. Well, I wish the built-in Ethernet ports weren't RJ45 but SFP+, but that's really the only thing I wish to change. YES. Jesus. The fact that the state of the art for SFP+ motherboards is basically still Denverton (C3758 etc) is embarrassing. You have all these "server" motherboards with consumer base-t standards... even if you have base-t to the workstations, surely your fancy SOHO/SMB setup will have a server closet where it would make sense to have SFP+ for the server, to punch down a bunch of individual base-T links... (and actually base-t has much more latency than SFP+ as well - measurable on NVMe drives etc) In theory this is something that OCP 2.0 mezzanine cards (like MZ31-AR0 uses) can do for you. Actually these are quite cheap because of the very limited surplus market for them, and you can get adapter cards to put them in PCIe slots if you want (the adapter cards are unusual enough they're not cheap, but like most pcie adapter cards they're not inherently expensive or electrically difficult). So you can put SFP+ on anything you want with a OCP 2.0 slot - but of course most of the asrock rack, supermicro, etc are all base-t with no OCP 2.0. Infuriating. (OCP 2.0 does make "traditional" IO area very difficult however, it eats a lot of space in that IO shield area, and this often has the side-effect that the CPU gets pushed over into the other side of the board where it starts overlapping pcie slots etc. There are reasons to not do it - and this is another problem brought on by the ATX layout. But then offer some SFP+ please - really SFP28 25gbit should be available at this point on things like the ROMED8 model imo.) And before someone says Minisforum MS-01... no ECC (which could be forgiven) and the chinese vendors are unfortunately quite poor in the support department in general. They are great at marketing via influencers etc but I have read a number of people say they've run them long-term and were upset about the support story. Which is a shame because on paper it's quite attractive - 12/13th gen are incredibly speedy (faster than AMD 7000 series) and most server workloads don't benefit that much from v-cache, a laptop CPU (again, ideally with ECC) with a couple SFP28 cards is more or less an ideal 25gb switch, can serve NVMe flash drives pretty fast, etc. It is quite desirable to have high per-thread performance in a homelab, especially when you are the sole user (4 people running 1 GB/s is not the same as 1 person running 4 GB/s). i3 7100 was actually an extremely interesting sku for this reason - 3.9 GHz dual-core (7350K is 4.2 GHz base, and 8350K/9350K were 4.2 GHz base 4C4T) is quite punchy, you never drop clocks due to AVX offset, and i3s supported ECC in this era. AM4 server boards weren't mature yet (X470D4U was one of the first good ones and it still took several years to stabilize fully) so the alternative was sandy bridge xeons etc, and the 7100 (while not a popular gaming CPU) absolutely destroyed that shit for homelab NAS builds, in both perf and efficiency. Again, it's kind of a tragedy that C3758 is the norm still - that's 8x 14nm e-cores with RDIMM support and onboard Intel QAT (of whatever gen). That is not fast at all, we are talking like sub-zen1 performance here probably, with no AVX, etc. (Unfortunately minisforum's AMD boards are not any better in the support department, and AMD segments ECC to the Pro laptop chips too, etc.) I think in practice the problem is the length of the SFP+ cage - it's noticeably longer than base-t. And that means either you're wasting space on your base-t boards, or you have to design a custom layout for SFP+, which is already a (truthfully) small/niche market etc. It is understandable, just unfortunate.