5 ms·

As an outsider to 'enterprise-grade' computing, I'm curious about situations where a high number of cores in a single processor would be superior to multiple pr

As an outsider to 'enterprise-grade' computing, I'm curious about situations where a high number of cores in a single processor would be superior to multiple processors with the same total energy draw sitting on a single motherboard? I can understand HPC applications where the high-speed interconnect on the chip would make a big difference. But in business applications where the cores are dedicated to running independent VMs, or are handling independent client requests, what is really gained? There would still be some benefits from a shared cache, but how large quantitatively would that be?

loeg 8y ago
NUMA. Latency between sockets is far higher than in a single socket. If your workload is truly wholly independent threads as you've described, then it's quite possible there is no benefit. (Although, sibling comments bring up good points about licensing fees.)
downrightmike 8y ago
When VMware charges per socket and not per core.
- NickNameNick 8y ago
  Big ditto for Oracle. probably others.
  - downrightmike 8y ago
    Realistically, it is easy to count cores. Some companies still count MAC addresses.
    - haikuginger 8y ago
      The way Oracle and VMWare bill is not bound by a technical limitation.
wmantly 8y ago
It has to do with memory. In server grade computers, each socket has memory local slots that it can read and write to very fast. Read this: https://en.wikipedia.org/wiki/Non-uniform_memory_access https://en.wikipedia.org/wiki/Non-uniform_memory_access
- Kubuxu 8y ago
  It is already the case with Thread Ripper processors. They have multiple NUMA nodes inside one socket.
  - Yizahi 8y ago
    Exactly the same case as single die Xeon architecture with 2 separate rings inside with different memory modules attached to each ring - https://images.anandtech.com/doci/9193/HaswellEPHCCdie_575px.png https://images.anandtech.com/doci/9193/HaswellEPHCCdie_575px...
  - devttyeu 8y ago
    It actually presents itself to the system as a single node: On a TR 1920x system: $ numactl --hardware available: 1 nodes (0) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 node 0 size: 32107 MB node 0 free: 20738 MB node distances: node 0 0: 10
    - bpye 8y ago
      I'm pretty sure you can change that, it should be a BIOS option [1]. [1] - https://www.anandtech.com/show/11697/the-amd-ryzen-threadripper-1950x-and-1920x-review/3 https://www.anandtech.com/show/11697/the-amd-ryzen-threadrip...
    - throwaway2048 8y ago
      windows is spectacularly poor at dealing with NUMA CPUS so threadripper is not displayed to the OS as NUMA.
      - 8y ago
vbezhenar 8y ago
If your cores are independent, you're blessed and can scale just by adding more cheap servers.
- xxs 8y ago
  Yeah, embarrassing parallel stuff is not very interesting. However when there is communication necessary - the length of the bus matters and having the dies next to each other does help a lot
pdpi 8y ago
I can see two answers to that. First is that a single-socket motherboard is still a simpler design to produce with all the advantages that entails. Second is that you’re allowed to stick two of these on a two-socket board for CPU-bound loads. Better density for when you have the thermal capacity to spare.
Shivetya 8y ago
I am on POWER8 at work, the wiki article [1] gives a great description of the advantages of many cores per chip though ours only has 6/12 cores. Part of our hardware configuration to migrate from POWER7 to POWER8 was to have 40g of memory per core available. I think POWER7 was 30g. We use this in the iSeries environment but we have pSeries machines with the same hardware running AIX/Oracle and POWER7 VMs running many *nix implementations. In my usage case, the core/thread count really helps DB2's SQL implementation as an iSeries is effectively a giant DB2 database with extras added on. Hence query engine (SQE/CQE see old doc [2] on our machine can make great use of many cores/threads. When serving data to intensive batch applications as well as thousands of warehouse users and double that through web services access to data is the name of the game. [1]https://en.wikipedia.org/wiki/POWER8 https://en.wikipedia.org/wiki/POWER8 [2] https://www.ibm.com/support/knowledgecenter/en/ssw_i5_54/rzajq/rzajqsqecqe.htm https://www.ibm.com/support/knowledgecenter/en/ssw_i5_54/rza... <- that is quite a few years old but describes the query engines available - CQE is 'legacy' and SQE is modern
- robbyt 8y ago
  Have you compared performance of the DB running in Linux on a property sized Intel/Xeon server? I've seen several mainframe companies dogmatically believing their sales rep their workload is special and needs a high-end system. But none of them I've talked to have actually tested for themselves.
dragontamer 8y ago
> As an outsider to 'enterprise-grade' computing, I'm curious about situations where a high number of cores in a single processor would be superior to multiple processors with the same total energy draw sitting on a single motherboard? Databases are the big one I'm aware of. Intel's L3 cache is truly unified. Intel's 28-core Skylake means that the L3 of a Database is TRULY 38.5MB. When any core requests data, it goes into the giant distributed L3 cache that all cores can access efficiently. AMD's L3 cache however is a network of 8MB chunks. Sure, there's 32MB of cache in its 32-core system, but any one core can only use 8MB of it effectively. In fact, pulling memory off of a "remote L3 cache" is slower (higher latency) than pulling it from RAM on the Threadripper / EPYC platform. (A remote L3 pull has to coordinate over infinity fabric and remain cohesive! So that means "invalidating" and waiting to become the "exclusive owner" before a core can start writing to a L3 cache line, well according to MESI cc-protocol. I know AMD uses something more complex and efficient... but my point is that cache-coherence has a cost that becomes clear in this case. ) Which doesn't bode well for any HPC application... but also for Databases (which will effectively be locked to 8MB per thread, with "poor sharing", at least compared to Xeon). Of course, "Databases" might be just the most common HPC application in the enterprise, that needs communication and coordination between threads.
- undersuit 8y ago
  >Intel's L3 cache is truly unified. Intel's 28-core Skylake means that the L3 of a Database is TRULY 38.5MB. When any core requests data, it goes into the giant distributed L3 cache that all cores can access efficiently. This is less true now. Intel's L3 cache is still all on one piece of monolithic silicon, unlike the 4 separate caches of the 4 separate dies on a 32-core TR. But the L3 slice for each core is now physically placed right next to the core and other slices are accessed through the ringbus or in Skylake and later, the mesh. Still faster than leaving the die and using AMD's Infinity Fabric, and a lot less complicated than wiring up all the cores for direct L3 access. https://www.anandtech.com/show/3922/intels-sandy-bridge-architecture-exposed/4 https://www.anandtech.com/show/3922/intels-sandy-bridge-arch...

loeg 8y ago

NUMA. Latency between sockets is far higher than in a single socket. If your workload is truly wholly independent threads as you've described, then it's quite possible there is no benefit. (Although, sibling comments bring up good points about licensing fees.)

downrightmike 8y ago

When VMware charges per socket and not per core.

NickNameNick 8y ago
Big ditto for Oracle. probably others.
- downrightmike 8y ago
  Realistically, it is easy to count cores. Some companies still count MAC addresses.
  - haikuginger 8y ago
    The way Oracle and VMWare bill is not bound by a technical limitation.

wmantly 8y ago

It has to do with memory. In server grade computers, each socket has memory local slots that it can read and write to very fast. Read this: https://en.wikipedia.org/wiki/Non-uniform_memory_access https://en.wikipedia.org/wiki/Non-uniform_memory_access

Kubuxu 8y ago
It is already the case with Thread Ripper processors. They have multiple NUMA nodes inside one socket.
- Yizahi 8y ago
  Exactly the same case as single die Xeon architecture with 2 separate rings inside with different memory modules attached to each ring - https://images.anandtech.com/doci/9193/HaswellEPHCCdie_575px.png https://images.anandtech.com/doci/9193/HaswellEPHCCdie_575px...
- devttyeu 8y ago
  It actually presents itself to the system as a single node: On a TR 1920x system: $ numactl --hardware available: 1 nodes (0) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 node 0 size: 32107 MB node 0 free: 20738 MB node distances: node 0 0: 10
  - bpye 8y ago
    I'm pretty sure you can change that, it should be a BIOS option [1]. [1] - https://www.anandtech.com/show/11697/the-amd-ryzen-threadripper-1950x-and-1920x-review/3 https://www.anandtech.com/show/11697/the-amd-ryzen-threadrip...
  - throwaway2048 8y ago
    windows is spectacularly poor at dealing with NUMA CPUS so threadripper is not displayed to the OS as NUMA.
    - 8y ago

vbezhenar 8y ago

If your cores are independent, you're blessed and can scale just by adding more cheap servers.

xxs 8y ago
Yeah, embarrassing parallel stuff is not very interesting. However when there is communication necessary - the length of the bus matters and having the dies next to each other does help a lot

pdpi 8y ago

I can see two answers to that. First is that a single-socket motherboard is still a simpler design to produce with all the advantages that entails. Second is that you’re allowed to stick two of these on a two-socket board for CPU-bound loads. Better density for when you have the thermal capacity to spare.

Shivetya 8y ago

I am on POWER8 at work, the wiki article [1] gives a great description of the advantages of many cores per chip though ours only has 6/12 cores. Part of our hardware configuration to migrate from POWER7 to POWER8 was to have 40g of memory per core available. I think POWER7 was 30g. We use this in the iSeries environment but we have pSeries machines with the same hardware running AIX/Oracle and POWER7 VMs running many *nix implementations. In my usage case, the core/thread count really helps DB2's SQL implementation as an iSeries is effectively a giant DB2 database with extras added on. Hence query engine (SQE/CQE see old doc [2] on our machine can make great use of many cores/threads. When serving data to intensive batch applications as well as thousands of warehouse users and double that through web services access to data is the name of the game. [1]https://en.wikipedia.org/wiki/POWER8 https://en.wikipedia.org/wiki/POWER8 [2] https://www.ibm.com/support/knowledgecenter/en/ssw_i5_54/rzajq/rzajqsqecqe.htm https://www.ibm.com/support/knowledgecenter/en/ssw_i5_54/rza... <- that is quite a few years old but describes the query engines available - CQE is 'legacy' and SQE is modern

robbyt 8y ago
Have you compared performance of the DB running in Linux on a property sized Intel/Xeon server? I've seen several mainframe companies dogmatically believing their sales rep their workload is special and needs a high-end system. But none of them I've talked to have actually tested for themselves.

dragontamer 8y ago

> As an outsider to 'enterprise-grade' computing, I'm curious about situations where a high number of cores in a single processor would be superior to multiple processors with the same total energy draw sitting on a single motherboard? Databases are the big one I'm aware of. Intel's L3 cache is truly unified. Intel's 28-core Skylake means that the L3 of a Database is TRULY 38.5MB. When any core requests data, it goes into the giant distributed L3 cache that all cores can access efficiently. AMD's L3 cache however is a network of 8MB chunks. Sure, there's 32MB of cache in its 32-core system, but any one core can only use 8MB of it effectively. In fact, pulling memory off of a "remote L3 cache" is slower (higher latency) than pulling it from RAM on the Threadripper / EPYC platform. (A remote L3 pull has to coordinate over infinity fabric and remain cohesive! So that means "invalidating" and waiting to become the "exclusive owner" before a core can start writing to a L3 cache line, well according to MESI cc-protocol. I know AMD uses something more complex and efficient... but my point is that cache-coherence has a cost that becomes clear in this case. ) Which doesn't bode well for any HPC application... but also for Databases (which will effectively be locked to 8MB per thread, with "poor sharing", at least compared to Xeon). Of course, "Databases" might be just the most common HPC application in the enterprise, that needs communication and coordination between threads.

undersuit 8y ago
>Intel's L3 cache is truly unified. Intel's 28-core Skylake means that the L3 of a Database is TRULY 38.5MB. When any core requests data, it goes into the giant distributed L3 cache that all cores can access efficiently. This is less true now. Intel's L3 cache is still all on one piece of monolithic silicon, unlike the 4 separate caches of the 4 separate dies on a 32-core TR. But the L3 slice for each core is now physically placed right next to the core and other slices are accessed through the ringbus or in Skylake and later, the mesh. Still faster than leaving the die and using AMD's Infinity Fabric, and a lot less complicated than wiring up all the cores for direct L3 access. https://www.anandtech.com/show/3922/intels-sandy-bridge-architecture-exposed/4 https://www.anandtech.com/show/3922/intels-sandy-bridge-arch...