9 ms·

> Xeon Platinum supports 8-socket machines. 6x memory channels x 8-sockets == 48x DIMMs for your massive 6TB RAM servers. Both AMD Epyc Rome and Ampere Altra c

> Xeon Platinum supports 8-socket machines. 6x memory channels x 8-sockets == 48x DIMMs for your massive 6TB RAM servers. Both AMD Epyc Rome and Ampere Altra can support 8TB of RAM in dual socket machines, so that spec isn't particularly impressive anymore. > Platinum is crazy priced for a crazy reason: its a building block to truly massive computers that only IBM matches. Meh. A dual-socket Rome system with 128 cores (256 threads), 8TB of RAM, and 128 PCIe 4.0 lanes is an enormous machine. A full 8 socket Platinum system is going to cost an order of magnitude (or two) more, and the performance improvement is going to be less than 2x. An 8-socket system is going to consume an enormous amount of power, cost an unbelievable amount of money, and the complex NUMA domain combined with Intel's interconnect isn't going to "just work" for almost any software, so you're likely going to put a lot of effort into making that system barely work. Intel Platinum is priced crazy because Intel is building monolithic processors (which means low yields) and Intel likes to have a substantial profit margin. Together, those mean high prices. At a certain point, you're better off investing in making your application work on an accelerator of some kind, in the form of a GPU or an FPGA or task-specific ASIC, rather than giving Intel all your money in the hopes of eeking out a marginal performance improvement. Alternatively, finding software that can scale to more than one machine. Maybe IBM has truly large systems that are worth considering. Intel doesn't right now, unless you're using some software that demands Intel for contractual reasons no one can argue with.

dragontamer 6y ago
But not with 6-channels per socket (48-channels total across an 8-socket computer). That's a heck of a lot of bandwidth. I was doing the specs from memory and conservatively: 128GB sticks across 48-channels. https://www.supermicro.com/en/products/system/7U/7088/SYS-7088B-TR4FT.cfm https://www.supermicro.com/en/products/system/7U/7088/SYS-70... This 8-way Supermicro system supports 192x DIMMs. So... 128GB sticks x 192 == 24TBs of RAM or so, maybe 48TBs. > At a certain point, you're better off investing in making your application work on an accelerator of some kind, in the form of a GPU or an FPGA or task-specific ASIC, rather than giving Intel all your money in the hopes of eeking out a marginal performance improvement. Alternatively, finding software that can scale to more than one machine. I mean, these 8-way systems are what? $500k ? Less than $1MM for sure. How many engineers do you need before a machine of that price is reasonable? Really, not that many. If you can solve a problem with a team of 10 engineers + 48 TBs of RAM, that's probably cheaper than trying to solve the same problem with 30 engineers with a cheaper computer. ----------- > An 8-socket system is going to consume an enormous amount of power, cost an unbelievable amount of money, and the complex NUMA domain combined with Intel's interconnect isn't going to "just work" for almost any software, so you're likely going to put a lot of effort into making that system barely work. NUMA scales better than RDMA / Ethernet / Infiniband. A lot, lot, LOT better. All communications are just memory-reads, and that's a real memory read. Not an "emulated memory read transmitted over Ethernet / RDMA pretending to be a memory read". You incur the NUMA penalty but that's almost certainly better than incurring a PCIe penalty.
- coder543 6y ago
  Sure, I agree it's a lot of bandwidth. But for that bandwidth to be used efficiently, the processes on each NUMA node need to almost exclusively limit themselves to memory attached to their node -- at which point, well-written software could probably do just as good spread out over several machines that are connected by multiple 100Gb network links, and then you saved two or three bajillion dollars. If you're heavily using the bandwidth over the NUMA interconnect, then you're not going to be using the memory bandwidth very effectively (and likely not really using the processor cores effectively), and that's when NVDIMMs like Optane Persistent Memory could give you large amounts of bulk memory storage on a smaller system. Or just use a number of Intel's new PCIe 4.0 Optane SSDs in a single machine in place of the extra memory and memory channels... the latency isn't the same as RAM, but it's much closer than traditional SSDs, and the bandwidth per SSD is like 7GB/s, which is impressive. It all depends on the application at hand, but there are solutions that cost a lot less than the Platinum machines for virtually every problem, in my opinion. I don't know... perhaps I'm too cynical of these cost-ineffective systems that just happen to be large, and I should be more impressed. > NUMA scales better than RDMA / Ethernet / Infiniband. A lot, lot, LOT better. Disagree.
  - dragontamer 6y ago
    > But for that bandwidth to be used efficiently, the processes on each NUMA node need to almost exclusively limit themselves to memory attached to their node -- at which point, well-written software could probably do just as good spread out over several machines that are connected by multiple 100Gb network links, and then you saved two or three bajillion dollars. Wait, so latency over NUMA is too much, but you're willing to incur a 100Gb network link? Intel's UPI is like 40GBps (Giga-BYTEs, not bits) at sub 500ns latency. 100Gb Ethernet is what? 10GBps in practice? 1/4th the bandwidth with 4x more latency (in the microseconds) or something? That's a PCIe latency penalty (x2, for the sender + the receiver). That's a penalty for electrical -> optical, and back again. Any latency, or bandwidth, bound problem is going to prefer a NUMA-link rather than 100Gbps over PCIe.
    - coder543 6y ago

dragontamer 6y ago

But not with 6-channels per socket (48-channels total across an 8-socket computer). That's a heck of a lot of bandwidth. I was doing the specs from memory and conservatively: 128GB sticks across 48-channels. https://www.supermicro.com/en/products/system/7U/7088/SYS-7088B-TR4FT.cfm https://www.supermicro.com/en/products/system/7U/7088/SYS-70... This 8-way Supermicro system supports 192x DIMMs. So... 128GB sticks x 192 == 24TBs of RAM or so, maybe 48TBs. > At a certain point, you're better off investing in making your application work on an accelerator of some kind, in the form of a GPU or an FPGA or task-specific ASIC, rather than giving Intel all your money in the hopes of eeking out a marginal performance improvement. Alternatively, finding software that can scale to more than one machine. I mean, these 8-way systems are what? $500k ? Less than $1MM for sure. How many engineers do you need before a machine of that price is reasonable? Really, not that many. If you can solve a problem with a team of 10 engineers + 48 TBs of RAM, that's probably cheaper than trying to solve the same problem with 30 engineers with a cheaper computer. ----------- > An 8-socket system is going to consume an enormous amount of power, cost an unbelievable amount of money, and the complex NUMA domain combined with Intel's interconnect isn't going to "just work" for almost any software, so you're likely going to put a lot of effort into making that system barely work. NUMA scales better than RDMA / Ethernet / Infiniband. A lot, lot, LOT better. All communications are just memory-reads, and that's a real memory read. Not an "emulated memory read transmitted over Ethernet / RDMA pretending to be a memory read". You incur the NUMA penalty but that's almost certainly better than incurring a PCIe penalty.

coder543 6y ago
Sure, I agree it's a lot of bandwidth. But for that bandwidth to be used efficiently, the processes on each NUMA node need to almost exclusively limit themselves to memory attached to their node -- at which point, well-written software could probably do just as good spread out over several machines that are connected by multiple 100Gb network links, and then you saved two or three bajillion dollars. If you're heavily using the bandwidth over the NUMA interconnect, then you're not going to be using the memory bandwidth very effectively (and likely not really using the processor cores effectively), and that's when NVDIMMs like Optane Persistent Memory could give you large amounts of bulk memory storage on a smaller system. Or just use a number of Intel's new PCIe 4.0 Optane SSDs in a single machine in place of the extra memory and memory channels... the latency isn't the same as RAM, but it's much closer than traditional SSDs, and the bandwidth per SSD is like 7GB/s, which is impressive. It all depends on the application at hand, but there are solutions that cost a lot less than the Platinum machines for virtually every problem, in my opinion. I don't know... perhaps I'm too cynical of these cost-ineffective systems that just happen to be large, and I should be more impressed. > NUMA scales better than RDMA / Ethernet / Infiniband. A lot, lot, LOT better. Disagree.
- dragontamer 6y ago
  > But for that bandwidth to be used efficiently, the processes on each NUMA node need to almost exclusively limit themselves to memory attached to their node -- at which point, well-written software could probably do just as good spread out over several machines that are connected by multiple 100Gb network links, and then you saved two or three bajillion dollars. Wait, so latency over NUMA is too much, but you're willing to incur a 100Gb network link? Intel's UPI is like 40GBps (Giga-BYTEs, not bits) at sub 500ns latency. 100Gb Ethernet is what? 10GBps in practice? 1/4th the bandwidth with 4x more latency (in the microseconds) or something? That's a PCIe latency penalty (x2, for the sender + the receiver). That's a penalty for electrical -> optical, and back again. Any latency, or bandwidth, bound problem is going to prefer a NUMA-link rather than 100Gbps over PCIe.
  - coder543 6y ago