9 ms·

We bought a $9000 single socket AMD (2.4 GHz) 40 core server from Dell and a $27,000 dual socket Intel (2.8 GHz) 100 core server from Dell. The AMD server is a

We bought a $9000 single socket AMD (2.4 GHz) 40 core server from Dell and a $27,000 dual socket Intel (2.8 GHz) 100 core server from Dell. The AMD server is at full load, using around 150W whereas the Intel server idles at 450W doing absolutely nothing and pushes up to 700W if we load it up. In hindsight, I regret not just buying 3 of the AMD servers for the same price. I had this fallacy that the Intel server would be magically more energy efficient, but that has turned out to be FUD and baloney. NB Power usage measured at the wall socket with voltage and current monitored PDUs.

manuelabeledo 4y ago
Sounds like there is something else going on there, since the Xeon CPU idle power consumption is nowhere near 200W [0] [0] https://www.anandtech.com/show/16594/intel-3rd-gen-xeon-scalable-review/5 https://www.anandtech.com/show/16594/intel-3rd-gen-xeon-scal...
- helf 4y ago
  Right? That is why I want to know more details instead of this vague garbage.
- helf 4y ago
  Not to mention the fact that "pushing up to 700W underload" is.. absolutely nothing since that means, ignoring everything else and just assuming CPU drawing that, the cores are only using 7w PER CORE. And this AMD system is using apparently a peak of 3.75w per core? really?
  - esperent 4y ago
    This comment made me realize I've never seen power per core specified. That seems like a great metric. Would calculating this be as simple as dividing peak usage by the number of cores? (Ignoring that modern CPUs have multiple core types)
    - toast0 4y ago
      Power per core is pretty variable though. If you've got a lightly threaded workload, you shouldn't be getting 40-52 cores per socket, but many of the cores could be at near 0w and some of the cores will use more power than they would on an all cores workload.
    - helf 4y ago
      It wouldn’t really. I only used that because the parent poster was making it sound like total power draws with the cores all pegged out. And if they are actually doing work and not just running full bore through a whole lot of NOPs then that would be a sorta average of power on each core. But as the other commenter said, the actual per core usage will vary /wildly/ even under high load. You would half to run something fairly specialized such as prime95, which can run entirely from cache, to get more accurate per core wattage values under same load maximum load. As it stands, my i9-13900K seems to have an idle “package” draw of around 10-12w and can peak at 360w under prime95 load. And that’s stock. I can make it ramp way past 360W with very little effort. But under “normal” desktop usage it happily sits well under 100w. So… shrugs. I would like to know more on this parent’s setup instead of “My server with literally 40% of the core count uses 1/6 of the power!” But we may not be so lucky.
      - aetherspawn 4y ago
        The load case is not 1:1 and the experience is only anecdotal, but I put a few more details about the load in another comment below. My commentary is only really about what we believed about AMD before making our purchase (internet forums like this one warn about energy use), how that wasn't really the case when we measured it on a real load, and why we 'feel' we would prefer to go the smaller and cheaper AMD server route next time.
        deleted 4y ago
        [deleted]
- paulmd 4y ago
  You have to remember that it's not just the CPU but also the IO (memory and pcie) and the chipset... collectively it's often called "platform power". Even if you don't have anything plugged into it, a big powerful chipset and lots of on-chip PCIe expansion still eats a decent amount of power. Also everything you plug into it will eat power, some 10gbe stuff and SAS controllers/etc are not lightweights in idle because it's not expected they'll be idle. Every HDD generally pulls about 5W when idle+spinning, even SSDs are a couple watts apiece. That said, yeah, that sounds way high for idle. ServeTheHome is a good place for stuff like this: > motherboard: X11SPM-F > memory: 6x 8GB RDIMM 2666Mt > kill-a-watt power on the wall at idle: 30..35Watts. https://forums.servethehome.com/index.php?threads/xeon-scalable-vs-epyc-idle-power-consumption.39037/ https://forums.servethehome.com/index.php?threads/xeon-scala... Single-socket though, but, another CPU probably doesn't add more than another 30 watts or so. Epyc actually tends to be worse on idle power because it has to run all its infinity fabric links at all times, if it wants to access that memory, and data movement is comparatively expensive. There were some early Milan/Zen3 epyc numbers that need to be disregarded due to problems with AMD's reference implementation motherboard, but, you can see from that thread it's not uncommon for Epyc to idle at 100W for a single-socket board. You can see from that thread that the consensus is Skylake-SP generally idles better. My own personal anecdote: I am tinkering with a dual-socket 2011-3 server right now (X10DRC-T4+) with a pair of 2697v3 and 2x4x32GB RDIMMs and quad 10gbase-t NICs, at idle I am around 75W at the wall... obviously Skylake-SP has a larger platform (it's the socket after mine) but it's not 350W bigger, guy has something wrong with his setup. It is definitely fair though that under load the Epycs are a lot more energy-efficient. Skylake-SP is still 14nm and Intel lingered there for far too long, 7nm is more than a full node better and Milan has higher IPC, higher clocks, and more cores. Ice Lake-SP was better than Skylake, but nobody bought it because it was years late to market and AMD was already better. So you are effectively comparing 2016-ish Intel stuff against AMD stuff that is 4+ years newer... but on the other hand that's the Intel stuff that is deployed right now, and people generally want to minimize the amount of generations in their clusters to make it easier to operate VMWare vMotion etc. /shruggy
helf 4y ago
While I don't doubt the AMD server with less than half the core count draws a lot less power, I would /like/ to know more details. Like what model/SKU servers? What are the actual model chips? How are they configured? blahblah Going "I got X and Y and with no further information X is so much better than Y!" online gets so fucking old. And I am genuinely curious. Not just being antagonistic. I am looking at some newer servers soon and have been debating Intel or AMD and I like seeing actual reallife results people have had.
deleted 4y ago
[deleted]
vel0city 4y ago
Which AMD chip is 40 cores?
- Tepix 4y ago
  There aren't any. There are some 48 core EPYC CPUs with a TDP between 225W and 360W. None of them are rated at 2.4Ghz. The EPYC 7351P 2.40Ghz has a mere 16 cores (32 threads) and is rated at 170W TDP.
deleted 4y ago
[deleted]
toast0 4y ago
Looking at wattage alone doesn't tell us very much; idle at 450W seems pretty broken, sure... but how much more traffic does the Intel at 700W than the AMD at 150W? With 2.5 the cores, I'd expect at least 2.5x the traffic?
- Retric 4y ago
  I’ve never seen linear scaling from a dual GPU setup. IMO you’re doing well to get 70% from the second CPU under full load.
  - toast0 4y ago
    Depends, sometimes you get better scaling from having 2x the cpu and 2x the memory (maybe 2x the disk) all in one box than if you have to split it up and have more cross-server connectivity. You wouldn't buy a dual socket box for 3x the money if you didn't think you'd get 3x the performance out of it, unless you really wanted to keep your server count down.
    - aetherspawn 4y ago
      For us, we only allocate ie 8 runners to a VM with 8 cores, which we pin to 8 cores on the actual server, so our focus is getting as many cores as possible to get as many runners as possible. We kind of expected that scaling HW with more CPUs instead of more entire rack systems would be more energy efficient, but I don’t think it makes too much of a difference compared to other factors like CPU/vendor architecture. FWIW our energy bill is $1000+ per month just from servers, and then several $k for various licensing on each host per year, so there was some sweet spot we were optimising for (low power, but less licensing cost via less racks). Our cost in the cloud would be $5k per month or more, so either way we still saved money this year.
    - Retric 4y ago
      My point was simply expecting linear scaling is a recipe for failure. Spending 3x as much for a 50% improved performance can be the best option.
fomine3 4y ago
* dual socket is power hungry for both platform * still 450W at idle isn't only due to dual CPU * 150W for full load, really?
- aetherspawn 4y ago
  Our use case is we're running around 30 GitHub Actions runners on VMs that are just building code in MATLAB. 'Full load' is => all the runners are occupied, CPU shows around 50% on the host. The idle case on the Intel server is RHEL OS running several Windows VMs that are doing nothing, and the loaded case is when we spin up a large VM with around 40 cores on that host and run something in MATLAB (essentially a similar job to what the runners would otherwise be running). The comparison is not completely accurate, but our experience was only anecdotal.
deleted 4y ago
[deleted]
TeeMassive 4y ago
I had similar experience in another job in a big corporation. It was obvious that AMD chips were way better and cost efficient. But management only wanted to pay for Intel because it's "Enterprise grade".
BoorishBears 4y ago
Some people would consider this comment "FUD and baloney" because of the number of confounding factors you're jumping over. You're measuring wall socket voltage on two different servers to make a commentary on CPU power usage, but that's including storage, power delivery efficiency, cooling solutions, etc You didn't describe what "we load it up" means for the Intel server, or how that compares to the AMD server, or what load is for that even (you're saying you keep the AMD server pegged at 100% CPU, or you have 100% of your expected load?). Presumably you didn't get a dual socket CPU for giggles, so the load isn't trivially comparable between both systems by either measure. If you're buying servers because of fallacies about entire brands, please leave the procurement to someone more experienced. That kind of stuff is more for building a gaming PC, server hardware is way too varied for "brand X is Y". You need to examine things at the SKU level.
- deleted 4y ago
  [deleted]

manuelabeledo 4y ago

Sounds like there is something else going on there, since the Xeon CPU idle power consumption is nowhere near 200W [0] [0] https://www.anandtech.com/show/16594/intel-3rd-gen-xeon-scalable-review/5 https://www.anandtech.com/show/16594/intel-3rd-gen-xeon-scal...

helf 4y ago
Right? That is why I want to know more details instead of this vague garbage.
helf 4y ago
Not to mention the fact that "pushing up to 700W underload" is.. absolutely nothing since that means, ignoring everything else and just assuming CPU drawing that, the cores are only using 7w PER CORE. And this AMD system is using apparently a peak of 3.75w per core? really?
- esperent 4y ago
  This comment made me realize I've never seen power per core specified. That seems like a great metric. Would calculating this be as simple as dividing peak usage by the number of cores? (Ignoring that modern CPUs have multiple core types)
  - toast0 4y ago
    Power per core is pretty variable though. If you've got a lightly threaded workload, you shouldn't be getting 40-52 cores per socket, but many of the cores could be at near 0w and some of the cores will use more power than they would on an all cores workload.
  - helf 4y ago
    It wouldn’t really. I only used that because the parent poster was making it sound like total power draws with the cores all pegged out. And if they are actually doing work and not just running full bore through a whole lot of NOPs then that would be a sorta average of power on each core. But as the other commenter said, the actual per core usage will vary /wildly/ even under high load. You would half to run something fairly specialized such as prime95, which can run entirely from cache, to get more accurate per core wattage values under same load maximum load. As it stands, my i9-13900K seems to have an idle “package” draw of around 10-12w and can peak at 360w under prime95 load. And that’s stock. I can make it ramp way past 360W with very little effort. But under “normal” desktop usage it happily sits well under 100w. So… shrugs. I would like to know more on this parent’s setup instead of “My server with literally 40% of the core count uses 1/6 of the power!” But we may not be so lucky.
    - aetherspawn 4y ago
      The load case is not 1:1 and the experience is only anecdotal, but I put a few more details about the load in another comment below. My commentary is only really about what we believed about AMD before making our purchase (internet forums like this one warn about energy use), how that wasn't really the case when we measured it on a real load, and why we 'feel' we would prefer to go the smaller and cheaper AMD server route next time.
      - deleted 4y ago
        [deleted]
paulmd 4y ago
You have to remember that it's not just the CPU but also the IO (memory and pcie) and the chipset... collectively it's often called "platform power". Even if you don't have anything plugged into it, a big powerful chipset and lots of on-chip PCIe expansion still eats a decent amount of power. Also everything you plug into it will eat power, some 10gbe stuff and SAS controllers/etc are not lightweights in idle because it's not expected they'll be idle. Every HDD generally pulls about 5W when idle+spinning, even SSDs are a couple watts apiece. That said, yeah, that sounds way high for idle. ServeTheHome is a good place for stuff like this: > motherboard: X11SPM-F > memory: 6x 8GB RDIMM 2666Mt > kill-a-watt power on the wall at idle: 30..35Watts. https://forums.servethehome.com/index.php?threads/xeon-scalable-vs-epyc-idle-power-consumption.39037/ https://forums.servethehome.com/index.php?threads/xeon-scala... Single-socket though, but, another CPU probably doesn't add more than another 30 watts or so. Epyc actually tends to be worse on idle power because it has to run all its infinity fabric links at all times, if it wants to access that memory, and data movement is comparatively expensive. There were some early Milan/Zen3 epyc numbers that need to be disregarded due to problems with AMD's reference implementation motherboard, but, you can see from that thread it's not uncommon for Epyc to idle at 100W for a single-socket board. You can see from that thread that the consensus is Skylake-SP generally idles better. My own personal anecdote: I am tinkering with a dual-socket 2011-3 server right now (X10DRC-T4+) with a pair of 2697v3 and 2x4x32GB RDIMMs and quad 10gbase-t NICs, at idle I am around 75W at the wall... obviously Skylake-SP has a larger platform (it's the socket after mine) but it's not 350W bigger, guy has something wrong with his setup. It is definitely fair though that under load the Epycs are a lot more energy-efficient. Skylake-SP is still 14nm and Intel lingered there for far too long, 7nm is more than a full node better and Milan has higher IPC, higher clocks, and more cores. Ice Lake-SP was better than Skylake, but nobody bought it because it was years late to market and AMD was already better. So you are effectively comparing 2016-ish Intel stuff against AMD stuff that is 4+ years newer... but on the other hand that's the Intel stuff that is deployed right now, and people generally want to minimize the amount of generations in their clusters to make it easier to operate VMWare vMotion etc. /shruggy

helf 4y ago

While I don't doubt the AMD server with less than half the core count draws a lot less power, I would /like/ to know more details. Like what model/SKU servers? What are the actual model chips? How are they configured? blahblah Going "I got X and Y and with no further information X is so much better than Y!" online gets so fucking old. And I am genuinely curious. Not just being antagonistic. I am looking at some newer servers soon and have been debating Intel or AMD and I like seeing actual reallife results people have had.

deleted 4y ago

[deleted]

vel0city 4y ago

Which AMD chip is 40 cores?

Tepix 4y ago
There aren't any. There are some 48 core EPYC CPUs with a TDP between 225W and 360W. None of them are rated at 2.4Ghz. The EPYC 7351P 2.40Ghz has a mere 16 cores (32 threads) and is rated at 170W TDP.

deleted 4y ago

[deleted]

toast0 4y ago

Looking at wattage alone doesn't tell us very much; idle at 450W seems pretty broken, sure... but how much more traffic does the Intel at 700W than the AMD at 150W? With 2.5 the cores, I'd expect at least 2.5x the traffic?

Retric 4y ago
I’ve never seen linear scaling from a dual GPU setup. IMO you’re doing well to get 70% from the second CPU under full load.
- toast0 4y ago
  Depends, sometimes you get better scaling from having 2x the cpu and 2x the memory (maybe 2x the disk) all in one box than if you have to split it up and have more cross-server connectivity. You wouldn't buy a dual socket box for 3x the money if you didn't think you'd get 3x the performance out of it, unless you really wanted to keep your server count down.
  - aetherspawn 4y ago
    For us, we only allocate ie 8 runners to a VM with 8 cores, which we pin to 8 cores on the actual server, so our focus is getting as many cores as possible to get as many runners as possible. We kind of expected that scaling HW with more CPUs instead of more entire rack systems would be more energy efficient, but I don’t think it makes too much of a difference compared to other factors like CPU/vendor architecture. FWIW our energy bill is $1000+ per month just from servers, and then several $k for various licensing on each host per year, so there was some sweet spot we were optimising for (low power, but less licensing cost via less racks). Our cost in the cloud would be $5k per month or more, so either way we still saved money this year.
  - Retric 4y ago
    My point was simply expecting linear scaling is a recipe for failure. Spending 3x as much for a 50% improved performance can be the best option.

fomine3 4y ago

* dual socket is power hungry for both platform * still 450W at idle isn't only due to dual CPU * 150W for full load, really?

aetherspawn 4y ago
Our use case is we're running around 30 GitHub Actions runners on VMs that are just building code in MATLAB. 'Full load' is => all the runners are occupied, CPU shows around 50% on the host. The idle case on the Intel server is RHEL OS running several Windows VMs that are doing nothing, and the loaded case is when we spin up a large VM with around 40 cores on that host and run something in MATLAB (essentially a similar job to what the runners would otherwise be running). The comparison is not completely accurate, but our experience was only anecdotal.

deleted 4y ago

[deleted]

TeeMassive 4y ago

I had similar experience in another job in a big corporation. It was obvious that AMD chips were way better and cost efficient. But management only wanted to pay for Intel because it's "Enterprise grade".

BoorishBears 4y ago

Some people would consider this comment "FUD and baloney" because of the number of confounding factors you're jumping over. You're measuring wall socket voltage on two different servers to make a commentary on CPU power usage, but that's including storage, power delivery efficiency, cooling solutions, etc You didn't describe what "we load it up" means for the Intel server, or how that compares to the AMD server, or what load is for that even (you're saying you keep the AMD server pegged at 100% CPU, or you have 100% of your expected load?). Presumably you didn't get a dual socket CPU for giggles, so the load isn't trivially comparable between both systems by either measure. If you're buying servers because of fallacies about entire brands, please leave the procurement to someone more experienced. That kind of stuff is more for building a gaming PC, server hardware is way too varied for "brand X is Y". You need to examine things at the SKU level.

deleted 4y ago
[deleted]