7 ms·

The Supermicro H11DSi Motherboard Mini-Review: The Sole Dual EPYC Solution

65 points by rubyn00bie 6y ago

metaphor 6y ago
First impression was single onboard VGA output and DE-9 serial port...typical enterprise. Second impression was damn does that first socket look like a peripheral hog. EDIT: Removed remark that erroneously referenced this comment[1] as a 2U chassis being applicable to the motherboard in question. [1] https://www.anandtech.com/comments/15783/the-supermicro-h11dsi-motherboard-review/700552 https://www.anandtech.com/comments/15783/the-supermicro-h11d...
- LIV2 6y ago
  The VGA output would be part of the IPMI solution for remote management, serial would be for out of band management too. Both are useful if you don't want to drive to the datacenter
  - rbanffy 6y ago
    For a very long time I joked that no self-respecting server had a VGA, keyboard or mouse ports. Serious metal should have serial ports for the operator console. That didn't age too well...
    - mshook 6y ago
      Well technically you can often ssh to the BMC and connect to the OS through an emulated serial link... Serial port is awesome to me just because you can copy/paste...
    - p_l 6y ago
      It gets better thanks to UEFI, since it provides serial-safe approach for system management. IBM PC compatible systems had no serial console and it shows in abominations like some Intel RAID controllers that required mouse to get through their graphical UIs. Even if you used one of the special "UI-over-serial" cards that were used on some PC servers, lack of support meant that it was perfectly normal for some option from to expect bitmapped graphics and want to use it :( With UEFI providing a basic UI framework that covers all that firmware needs (and which still allows for fancy looking UI if you connect to framebuffer) serial console is back in use :)
- robin_reala 6y ago
  That’s a completely different chassis, as seen by the “Integrated Board: Super H12DST-B” photo. The review is of the H11DSi, which is designed for (big) tower cases.
  - metaphor 6y ago
    Thanks for pointing out that missed detail on my part.
- p_l 6y ago
  majority of BMC solutions default to VGA connection, and any good one will have a serial port - either DE-9 or RJ style. It's much, much easier to standardize on VGA-based iKVMs than otherwise, and this drives the "let's use VGA" movement. (Only real difference would be switching to DisplayPort, and this doesn't bring any benefits either...)
  - metaphor 6y ago
    Completely aware of the enterprise implication, but it's somewhat of a disappoint that with all that peripheral support, they could have at least dropped a courtesy single link DVI-I interface instead. It just smells like a decision driven to explicitly avoid paying out licensing royalties.
    - p_l 6y ago
      And pray tell, what would drive the digital part of the DVI? The video source has only analog VGA output available, so you can technically use a DVI-A connector, but pretty much nothing used them. And while DVI/HDMI signaling is pretty much VGA-but-analog, recovering digital stream out of analog VGA signal would make no sense.
greendave 6y ago
This feels a bit like they just took an existing Intel design and tried to adapt it for AMD. Nothing wrong with that per se - by all accounts it's a good solid board - but it leads to weird choices like not using 56 of 64 PCIe lanes on the second CPU.
- xdxdx 6y ago
  There are some other potential reasons: - Simpler PCB routing (= fewer layers = cheaper board) only using lanes from the CPU closer to the slots - Slots still work when 2nd CPU is not installed without extra chips - Board is already not designed around heavy I/O use cases given lack of PCIe 4.0 support This definitely feels like a board that's mostly intended for compute-intensive workloads that do not need much PCIe bandwidth.
- rumanator 6y ago
  > (...) but it leads to weird choices like not using 56 of 64 PCIe lanes on the second CPU. The article mentions that out of a total of 128 PCI lanes, 64 are used for CPU-to-CPU communication and only 8 are used for an external PCIe device. To be fair, a user that feels the need to cram two EPYC processors on the same motherboard is someone who is willing to pay a premium for CPU-bound computations.
deleted 6y ago
[deleted]
chx 6y ago
Anandtech. I called this guy out on Twitter eventually challenging him to show me any article (in this PC area) of theirs and I will find mistakes. I offered my editorial services for one dollar. I've been, long ago, one of the editors of the largest computer monthly in Hungary and was on the editorial board. This wouldn't have been my first rodeo. In this particular article there's not a lot of technical explanation so it's not easy to find a mistake but it's not so hard either: the GIGABYTE MZ31-AR0 is seriously outdated, they now sell the MZ32-AR0 which is now PCIe 4.0 and they missed from the list the Tomcat HX S8030 which is also PCIe 4.0. They did not, in fact, list any PCIe 4.0 boards. It's just sloppy.
- threeseed 6y ago
  The boards they listed e.g. MZ31-AR0 they have reviews for. Cross linking in your site improves traffic and SEO which is far important to a website than whether they listed a PCIe 4.0 board or not.
  - chx 6y ago
    They listed the ASUS KNPA-U16 which they also didn't review...
- wtallis 6y ago
  > I called this guy out on Twitter eventually challenging him to show me any article (in this PC area) of theirs and I will find mistakes. I offered my editorial services for one dollar. I've been, long ago, one of the editors of the largest computer monthly in Hungary and was on the editorial board. This wouldn't have been my first rodeo. Do you honestly think anyone would hire you with an approach like that? You'd have to pay people to put up with that attitude.
  - chx 6y ago
    Sigh. People missed the word "eventually". The discussion was more pleasant. I have pointed out some mistake in an article politely and offered to help. He was quite adamant they need no help and was, in fact, quite arrogant and that's when I told him that a) everyone knows Anandtech is sloppy, actually linking a HN comment form a few months back b) I challenged him to show me anything that I can't find a mistake in.
lazyjones 6y ago
Not even a memory bandwidth benchmark? Disappointing... It would be interesting to see how the 2 x N cores can deal with 16 memory sockets only and what the bandwidth of the low core vs. the high core CPU looks like.
nullc 6y ago
I have this board with 7742s. I like it generally, though most of my affection is due to the CPUs. I wish it had PCI-E 4. I've found it to be unreliable with memory running faster than 2933MHz but that might be the memory I was using. When I did have memory corruption problems running at higher memory speeds I didn't get a single ECC error, which I found surprising and concerning. I also had issues with Kernel 5.4 not booting at all ( https://bugzilla.redhat.com/show_bug.cgi?id=1783733 https://bugzilla.redhat.com/show_bug.cgi?id=1783733 ) which I worked around by adding edac_report=off (I'm not sure if later kernels don't have this issue, because I've left the setting in, since taking it down and rebooting to test it is a bit of a pain-- as is typical with server boards with lots of cores and ram it takes many minutes to post)
- jnwatson 6y ago
  "When I did have memory corruption problems running at higher memory speeds I didn't get a single ECC error, which I found surprising and concerning." How is this possible?
  - jeffbee 6y ago
    It's possible because many motherboards and even integrated systems fail to setup EDAC properly at boot time.
- arminiusreturns 6y ago
  Having built a few quad/dual cpu AMD Opteron systems since ~2013 with supermicro boards, I can tell you they are extremely sensitive to RAM brands/models and variations. Sometimes even RAM in the verified supermicro list would have issues. On a side note, it was about then that I foresaw AMD's move towards high core counts and I love what they have been doing both on desktop and in the server space. Honestly though, I think if I were to try to do a newer amd server I might try to push a threadripper if the power profile fit the DC/colo instead of an epyc mostly because of the lanes and the fact that anything I'm doing on a system that powerful is going to have a 40gb+ mellanox and often in combination with GPUs.
  - 6y ago
ShamelessC 6y ago
I've admittedly never actually considered the dis/advantages of running two CPU's on one board. Is it correct that you can run two different processors with these? How much of a speedup does it give you? Is the operating system designed to take full advantage of both by default? Why don't gaming builds use these? Too much cost?
- posix_me_less 6y ago
  The same processors should be used, I would be surprised if different one would work together. The processor model has to support two socket operation, those are more expensive. Speedup happens only if the running system benefits from more cores. The expected speedup is <2. Operating systems work with this, but Windows may have some weird problems with many core processors. Gaming builds usually cannot take advantage of so many cores. Even if the machine is running the game and streaming, single 6-core or 8-core processor is enough.
  - rbanffy 6y ago
    > The same processors should be used This is an interesting thing. I should be possible to be smarter about different timings. ARM has very mature asymmetric core support for quite some time now. Intel will soon launch their own in the mobile space. > Gaming builds usually cannot take advantage of so many cores Indeed. Games have long been limited in their support for multiple cores. It's a bit surprising because consumer Windows, while it took its time to be able to support multiple sockets, has been able to support them for a very long time now.
    - rcxdude 6y ago
      It's mostly that the kind of CPU work games are doing is quite hard to parallelise (physics simulations, NPC 'AI', loading and decompressing model data). It's a lot of fairly complex code and getting a synchronisation strategy which actually produces speedup is hard. And when it is done it's usually still limited to e.g. 1 thread doing physics, 1-2 threads doing rendering (it took a while for graphics APIs to support any kind of multi-threaded rendering), etc so it only scales to 3-4 cores max (and there are games which do this, so core count is not completely irrelevant).
tw04 6y ago
It's extremely frustrating that none of the vendors are taking advantage of the built-in 10Gbe functionality of the new AMD chips. I've seen the same with the embedded options that compete with the Xeon-D. If anyone happens to work in that specific portion of the industry and can comment - I'd be curious if there's a reason other than motherboard mfgs just don't want to invest the time and effort so they slap a 1Gbe intel NIC on the mobo and call it a day. Little stuff like that makes the AMD option (which SHOULD be cheaper) not cheaper for a lot of applications because now you've got server + NIC vs. just the server from intel.
- kentonv 6y ago
  I recently researched EPYC 7002 boards available via retail channels and found four options: Supermicro, Asrock, Tyan, and Gigabyte. Of these, Asrock and Tyan both offer 10Gbe options. I ended up buying the Asrock board with 10Gbe from Newegg, but haven't hooked it up yet. https://www.asrockrack.com/general/productdetail.asp?Model=EPYCD8-2T https://www.asrockrack.com/general/productdetail.asp?Model=E... https://www.tyan.com/Motherboards_S8030_S8030GM4NE-2T https://www.tyan.com/Motherboards_S8030_S8030GM4NE-2T Though now that I look at it, the ASRock board is advertising "10G base-T by Intel® X550" -- is this different from the "built-in" functionality you mention?
  - tw04 6y ago
    Yup. AMD has an on-die 10gbe controller that would show up as AMD. Anything that says Intel/Broadcom/realtek/etc are off chip and something the MFG put on the motherboard which adds cost.
    - kentonv 6y ago
      Hmm. I'm speculating, but I'd guess that tossing on the same external controller that they've put on a million other motherboards in the past probably actually saves cost, vs. designing something unique to expose the on-die controller. I wonder what the performance impact is, though. I'd guess (very naively) that an on-die controller would perform better.
xVedun 6y ago
So I was looking at the H12DSU-iN(https://www.supermicro.com/en/products/motherboard/H12DSU-iN https://www.supermicro.com/en/products/motherboard/H12DSU-iN) which would seem to be another option. But it looks like the article implies that this would only be available to buy through getting an entire server with it. Although it looks like it could be possible that they sell it if you ask. The server they are going to release is here: https://www.supermicro.com/en/Aplus/system/1U/1024/AS-1024US-TRT.cfm https://www.supermicro.com/en/Aplus/system/1U/1024/AS-1024US...

65 points by rubyn00bie 6y ago

First impression was single onboard VGA output and DE-9 serial port...typical enterprise. Second impression was damn does that first socket look like a peripheral hog. EDIT: Removed remark that erroneously referenced this comment[1] as a 2U chassis being applicable to the motherboard in question. [1] https://www.anandtech.com/comments/15783/the-supermicro-h11dsi-motherboard-review/700552 https://www.anandtech.com/comments/15783/the-supermicro-h11d...

LIV2 6y ago
The VGA output would be part of the IPMI solution for remote management, serial would be for out of band management too. Both are useful if you don't want to drive to the datacenter
- rbanffy 6y ago
  For a very long time I joked that no self-respecting server had a VGA, keyboard or mouse ports. Serious metal should have serial ports for the operator console. That didn't age too well...
  - mshook 6y ago
    Well technically you can often ssh to the BMC and connect to the OS through an emulated serial link... Serial port is awesome to me just because you can copy/paste...
  - p_l 6y ago
    It gets better thanks to UEFI, since it provides serial-safe approach for system management. IBM PC compatible systems had no serial console and it shows in abominations like some Intel RAID controllers that required mouse to get through their graphical UIs. Even if you used one of the special "UI-over-serial" cards that were used on some PC servers, lack of support meant that it was perfectly normal for some option from to expect bitmapped graphics and want to use it :( With UEFI providing a basic UI framework that covers all that firmware needs (and which still allows for fancy looking UI if you connect to framebuffer) serial console is back in use :)
robin_reala 6y ago
That’s a completely different chassis, as seen by the “Integrated Board: Super H12DST-B” photo. The review is of the H11DSi, which is designed for (big) tower cases.
- metaphor 6y ago
  Thanks for pointing out that missed detail on my part.
p_l 6y ago
majority of BMC solutions default to VGA connection, and any good one will have a serial port - either DE-9 or RJ style. It's much, much easier to standardize on VGA-based iKVMs than otherwise, and this drives the "let's use VGA" movement. (Only real difference would be switching to DisplayPort, and this doesn't bring any benefits either...)
- metaphor 6y ago
  Completely aware of the enterprise implication, but it's somewhat of a disappoint that with all that peripheral support, they could have at least dropped a courtesy single link DVI-I interface instead. It just smells like a decision driven to explicitly avoid paying out licensing royalties.
  - p_l 6y ago
    And pray tell, what would drive the digital part of the DVI? The video source has only analog VGA output available, so you can technically use a DVI-A connector, but pretty much nothing used them. And while DVI/HDMI signaling is pretty much VGA-but-analog, recovering digital stream out of analog VGA signal would make no sense.

greendave 6y ago

This feels a bit like they just took an existing Intel design and tried to adapt it for AMD. Nothing wrong with that per se - by all accounts it's a good solid board - but it leads to weird choices like not using 56 of 64 PCIe lanes on the second CPU.

xdxdx 6y ago
There are some other potential reasons: - Simpler PCB routing (= fewer layers = cheaper board) only using lanes from the CPU closer to the slots - Slots still work when 2nd CPU is not installed without extra chips - Board is already not designed around heavy I/O use cases given lack of PCIe 4.0 support This definitely feels like a board that's mostly intended for compute-intensive workloads that do not need much PCIe bandwidth.
rumanator 6y ago
> (...) but it leads to weird choices like not using 56 of 64 PCIe lanes on the second CPU. The article mentions that out of a total of 128 PCI lanes, 64 are used for CPU-to-CPU communication and only 8 are used for an external PCIe device. To be fair, a user that feels the need to cram two EPYC processors on the same motherboard is someone who is willing to pay a premium for CPU-bound computations.

deleted 6y ago

[deleted]

chx 6y ago

Anandtech. I called this guy out on Twitter eventually challenging him to show me any article (in this PC area) of theirs and I will find mistakes. I offered my editorial services for one dollar. I've been, long ago, one of the editors of the largest computer monthly in Hungary and was on the editorial board. This wouldn't have been my first rodeo. In this particular article there's not a lot of technical explanation so it's not easy to find a mistake but it's not so hard either: the GIGABYTE MZ31-AR0 is seriously outdated, they now sell the MZ32-AR0 which is now PCIe 4.0 and they missed from the list the Tomcat HX S8030 which is also PCIe 4.0. They did not, in fact, list any PCIe 4.0 boards. It's just sloppy.

threeseed 6y ago
The boards they listed e.g. MZ31-AR0 they have reviews for. Cross linking in your site improves traffic and SEO which is far important to a website than whether they listed a PCIe 4.0 board or not.
- chx 6y ago
  They listed the ASUS KNPA-U16 which they also didn't review...
wtallis 6y ago
> I called this guy out on Twitter eventually challenging him to show me any article (in this PC area) of theirs and I will find mistakes. I offered my editorial services for one dollar. I've been, long ago, one of the editors of the largest computer monthly in Hungary and was on the editorial board. This wouldn't have been my first rodeo. Do you honestly think anyone would hire you with an approach like that? You'd have to pay people to put up with that attitude.
- chx 6y ago
  Sigh. People missed the word "eventually". The discussion was more pleasant. I have pointed out some mistake in an article politely and offered to help. He was quite adamant they need no help and was, in fact, quite arrogant and that's when I told him that a) everyone knows Anandtech is sloppy, actually linking a HN comment form a few months back b) I challenged him to show me anything that I can't find a mistake in.

lazyjones 6y ago

Not even a memory bandwidth benchmark? Disappointing... It would be interesting to see how the 2 x N cores can deal with 16 memory sockets only and what the bandwidth of the low core vs. the high core CPU looks like.

nullc 6y ago

I have this board with 7742s. I like it generally, though most of my affection is due to the CPUs. I wish it had PCI-E 4. I've found it to be unreliable with memory running faster than 2933MHz but that might be the memory I was using. When I did have memory corruption problems running at higher memory speeds I didn't get a single ECC error, which I found surprising and concerning. I also had issues with Kernel 5.4 not booting at all ( https://bugzilla.redhat.com/show_bug.cgi?id=1783733 https://bugzilla.redhat.com/show_bug.cgi?id=1783733 ) which I worked around by adding edac_report=off (I'm not sure if later kernels don't have this issue, because I've left the setting in, since taking it down and rebooting to test it is a bit of a pain-- as is typical with server boards with lots of cores and ram it takes many minutes to post)

jnwatson 6y ago
"When I did have memory corruption problems running at higher memory speeds I didn't get a single ECC error, which I found surprising and concerning." How is this possible?
- jeffbee 6y ago
  It's possible because many motherboards and even integrated systems fail to setup EDAC properly at boot time.
arminiusreturns 6y ago
Having built a few quad/dual cpu AMD Opteron systems since ~2013 with supermicro boards, I can tell you they are extremely sensitive to RAM brands/models and variations. Sometimes even RAM in the verified supermicro list would have issues. On a side note, it was about then that I foresaw AMD's move towards high core counts and I love what they have been doing both on desktop and in the server space. Honestly though, I think if I were to try to do a newer amd server I might try to push a threadripper if the power profile fit the DC/colo instead of an epyc mostly because of the lanes and the fact that anything I'm doing on a system that powerful is going to have a 40gb+ mellanox and often in combination with GPUs.
- 6y ago

ShamelessC 6y ago

I've admittedly never actually considered the dis/advantages of running two CPU's on one board. Is it correct that you can run two different processors with these? How much of a speedup does it give you? Is the operating system designed to take full advantage of both by default? Why don't gaming builds use these? Too much cost?

posix_me_less 6y ago
The same processors should be used, I would be surprised if different one would work together. The processor model has to support two socket operation, those are more expensive. Speedup happens only if the running system benefits from more cores. The expected speedup is <2. Operating systems work with this, but Windows may have some weird problems with many core processors. Gaming builds usually cannot take advantage of so many cores. Even if the machine is running the game and streaming, single 6-core or 8-core processor is enough.
- rbanffy 6y ago
  > The same processors should be used This is an interesting thing. I should be possible to be smarter about different timings. ARM has very mature asymmetric core support for quite some time now. Intel will soon launch their own in the mobile space. > Gaming builds usually cannot take advantage of so many cores Indeed. Games have long been limited in their support for multiple cores. It's a bit surprising because consumer Windows, while it took its time to be able to support multiple sockets, has been able to support them for a very long time now.
  - rcxdude 6y ago
    It's mostly that the kind of CPU work games are doing is quite hard to parallelise (physics simulations, NPC 'AI', loading and decompressing model data). It's a lot of fairly complex code and getting a synchronisation strategy which actually produces speedup is hard. And when it is done it's usually still limited to e.g. 1 thread doing physics, 1-2 threads doing rendering (it took a while for graphics APIs to support any kind of multi-threaded rendering), etc so it only scales to 3-4 cores max (and there are games which do this, so core count is not completely irrelevant).

tw04 6y ago

It's extremely frustrating that none of the vendors are taking advantage of the built-in 10Gbe functionality of the new AMD chips. I've seen the same with the embedded options that compete with the Xeon-D. If anyone happens to work in that specific portion of the industry and can comment - I'd be curious if there's a reason other than motherboard mfgs just don't want to invest the time and effort so they slap a 1Gbe intel NIC on the mobo and call it a day. Little stuff like that makes the AMD option (which SHOULD be cheaper) not cheaper for a lot of applications because now you've got server + NIC vs. just the server from intel.

kentonv 6y ago
I recently researched EPYC 7002 boards available via retail channels and found four options: Supermicro, Asrock, Tyan, and Gigabyte. Of these, Asrock and Tyan both offer 10Gbe options. I ended up buying the Asrock board with 10Gbe from Newegg, but haven't hooked it up yet. https://www.asrockrack.com/general/productdetail.asp?Model=EPYCD8-2T https://www.asrockrack.com/general/productdetail.asp?Model=E... https://www.tyan.com/Motherboards_S8030_S8030GM4NE-2T https://www.tyan.com/Motherboards_S8030_S8030GM4NE-2T Though now that I look at it, the ASRock board is advertising "10G base-T by Intel® X550" -- is this different from the "built-in" functionality you mention?
- tw04 6y ago
  Yup. AMD has an on-die 10gbe controller that would show up as AMD. Anything that says Intel/Broadcom/realtek/etc are off chip and something the MFG put on the motherboard which adds cost.
  - kentonv 6y ago
    Hmm. I'm speculating, but I'd guess that tossing on the same external controller that they've put on a million other motherboards in the past probably actually saves cost, vs. designing something unique to expose the on-die controller. I wonder what the performance impact is, though. I'd guess (very naively) that an on-die controller would perform better.

xVedun 6y ago

So I was looking at the H12DSU-iN(https://www.supermicro.com/en/products/motherboard/H12DSU-iN https://www.supermicro.com/en/products/motherboard/H12DSU-iN) which would seem to be another option. But it looks like the article implies that this would only be available to buy through getting an entire server with it. Although it looks like it could be possible that they sell it if you ask. The server they are going to release is here: https://www.supermicro.com/en/Aplus/system/1U/1024/AS-1024US-TRT.cfm https://www.supermicro.com/en/Aplus/system/1U/1024/AS-1024US...