4 ms·

Yeah, it makes a lot of sense to go with single socket servers unless you can't scale horizontally (e.g. database server). Why deal with the complexity when you

by eloff 5y ago

Yeah, it makes a lot of sense to go with single socket servers unless you can't scale horizontally (e.g. database server). Why deal with the complexity when you can just side step it.

dragontamer 5y ago
Why would you switch from a 100GBps NUMA connection (800 gigabits per second) over NUMA fabric into a 10 Gbps Ethernet fabric? If you are scaling horizontally, NUMA is the superior fabric than Ethernet or Infiniband (100Gbps) Horizontal scaling seems to favor NUMA. 1000 chips over Ethernet is less efficient than 500 dual socket nodes over Ethernet. Anything you can do over Ethernet seems easier and cheaper over NUMA instead.
- wmf 5y ago
  This is correct if your software is NUMA-optimized (or if auto-NUMA works well for you) but if it isn't you can end up with slowdowns.
  - dragontamer 5y ago
    Surely that can be fixed with just a well placed numactl command to set node affinity and CPU affinity. The root article is discussing rewriting code to fit on FPGAs. If NUMA is too complex then... I dunno. The FPGA argument seems dead on arrival.
- kortilla 5y ago
  Your scaling architecture sucks if it depends on that kind of throughput. If you need that you’ve only can kicked your way to more capacity without a real scaling fix.
  - dragontamer 5y ago
    Depend on? Heavens no. Dual socket has numerous advantages in density and rack space. The fact that performance is better is pretty much icing on the cake. It's easier to manage 500 dual socket servers than 1000 single socket servers. Less equipment, higher utilization of parts, etc. Etc. To suggest dual socket NUMA is going away is... just very unlikely to me. I don't see what the benefits would be at all. Not just performance, but also routine maintenance issues (power, Ethernet, local storage, etc etc)
- eloff 5y ago
  I'm talking mostly abour scaling things like app servers where they might not need any communication. But in general if you can't scale horizontally at 10 gbps, you're in for a world of hurt. Numa gets you to 8x scale at best on very expensive very exotic hardware. And then you hit the wall.
  - dragontamer 5y ago
    I'm mostly talking about 2 socket servers, which are IIRC more common than even single socket servers. Dual socket is a cheap, easy, and common. If only to recycle fans, power supplies and racks, it seems useful.
    - eloff 5y ago
      And single socket is equally cheap, except it takes twice the rack space - but it also gives you redundancy. One server can fail and you can carry on. The advantage of memory bandwidth vs Ethernet for scaling to x2 really doesn't matter. If it did, you're not horizontally scalable and at best you buy a little time before you hit the wall. If the price difference isn't much, I would heavily prefer single socket.

by eloff 5y ago

Yeah, it makes a lot of sense to go with single socket servers unless you can't scale horizontally (e.g. database server). Why deal with the complexity when you can just side step it.

dragontamer 5y ago

Why would you switch from a 100GBps NUMA connection (800 gigabits per second) over NUMA fabric into a 10 Gbps Ethernet fabric? If you are scaling horizontally, NUMA is the superior fabric than Ethernet or Infiniband (100Gbps) Horizontal scaling seems to favor NUMA. 1000 chips over Ethernet is less efficient than 500 dual socket nodes over Ethernet. Anything you can do over Ethernet seems easier and cheaper over NUMA instead.

wmf 5y ago
This is correct if your software is NUMA-optimized (or if auto-NUMA works well for you) but if it isn't you can end up with slowdowns.
- dragontamer 5y ago
  Surely that can be fixed with just a well placed numactl command to set node affinity and CPU affinity. The root article is discussing rewriting code to fit on FPGAs. If NUMA is too complex then... I dunno. The FPGA argument seems dead on arrival.
kortilla 5y ago
Your scaling architecture sucks if it depends on that kind of throughput. If you need that you’ve only can kicked your way to more capacity without a real scaling fix.
- dragontamer 5y ago
  Depend on? Heavens no. Dual socket has numerous advantages in density and rack space. The fact that performance is better is pretty much icing on the cake. It's easier to manage 500 dual socket servers than 1000 single socket servers. Less equipment, higher utilization of parts, etc. Etc. To suggest dual socket NUMA is going away is... just very unlikely to me. I don't see what the benefits would be at all. Not just performance, but also routine maintenance issues (power, Ethernet, local storage, etc etc)
eloff 5y ago
I'm talking mostly abour scaling things like app servers where they might not need any communication. But in general if you can't scale horizontally at 10 gbps, you're in for a world of hurt. Numa gets you to 8x scale at best on very expensive very exotic hardware. And then you hit the wall.
- dragontamer 5y ago
  I'm mostly talking about 2 socket servers, which are IIRC more common than even single socket servers. Dual socket is a cheap, easy, and common. If only to recycle fans, power supplies and racks, it seems useful.
  - eloff 5y ago
    And single socket is equally cheap, except it takes twice the rack space - but it also gives you redundancy. One server can fail and you can carry on. The advantage of memory bandwidth vs Ethernet for scaling to x2 really doesn't matter. If it did, you're not horizontally scalable and at best you buy a little time before you hit the wall. If the price difference isn't much, I would heavily prefer single socket.