4 ms·

> that $50 saved has slowed down $100M worth of stuff, sometimes by 50% or more. But... that's not the hosting provider's problem Provided their clients don't

> that $50 saved has slowed down $100M worth of stuff, sometimes by 50% or more. But... that's not the hosting provider's problem Provided their clients don't figure out what's going on and move to another provider. If the client has millions of dollars riding on that infrastructure, they'd likely be doing enough performance-monitoring to notice this kind of thing, no? > Azure does this, and they don't let you turn it off Azure really don't offer any never-throttled instances? What about their dedicated hosts?

jiggawatts 5y ago
> Provided their clients don't figure out what's going on I'm always pleasantly surprised when I see power management correctly configured at a client precisely because it is so rarely done correctly. Even if cloud vendors do fix this one thing, there's about a dozen more such issues where the suppliers' interests don't line up with the downstream consumers' interests. > Azure really don't offer any never-throttled instances? What about their dedicated hosts? I don't believe so. For example, their highest performing single node is the HBv3, with 2x AMD EPYC 7V13 processors for a total of 120 cores / 240 threads. On this page: https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/hpc/hbv3-series-overview https://docs.microsoft.com/en-us/azure/virtual-machines/work... There is a box that says: Nodes per Socket (NPS) = 2 L3 as NUMA = Disabled NUMA domains within VM OS = 4 C-states = Enabled The technical name for CPU power management is called "C-states". In other words, they've left it on the default (enabled) on a machine that they charge $40K/year for!
- maayank 5y ago
  > I'm always pleasantly surprised when I see power management correctly configured at a client precisely because it is so rarely done correctly. Let's say I have Linux instances in the cloud. What would be the steps to take for proper power management?
  - jiggawatts 5y ago
    As far as I know, there are no steps you can take if you're running your instance in a virtual machine. Only the hypervisor host and/or the firmware can control the processor power and sleep states.
- cwizou 5y ago
  TL;DR: C-States improve performance. I'm not sure calling C-States throttling is completely accurate, at least not in the way that you imply. Modern CPUs have multiple mechanisms that can be called throttling in some way, but C-States are the least offensive and in modern CPUs actually increase performance. There's a good basic explanation here [1] but in short : - C-States is about idle cores, and powering them down. You can actually configure how deep you can power them down. Waking up does have a tiny cost in latency, yes, but it's mostly irrelevant (your core spends most of it's time waiting for data). - P-States are what I would call "ok" throttling, those happen when non-idle, modern cores can scale their frequency up and down based on how much workload they have. Again, in theory purely about power saving and close to 0 impact on performance. This was simple, and used to be just about power saving, disabling was ok. But things changed massively about 10+ years ago with the introductions of various Turbo mechanisms. Turbo is about looking at your CPU as a whole (and not per core), and maximising performance of the loaded cores by pushing their frequency, using the power budget that used to be allocated to the idle cores (that are now powered off). The way its done is by adding a few extra P-states that push the frequency a few hundred MHz. The more "C-State-down" cores you have, the higher the frequency of the working cores is pushed. Extreme example : you have a 28 cores CPUs running a single thread workload on one core only, with 27 idle cores. With C-States disabled, you lose Turbo and are stuck with the single core running at the "non-Turbo" frequency. With C-States enabled, because you idled the non used cores, your working core gets higher clock speeds and higher performances. (Things get more complex with AVX2 because those units have a large power consumption, and this is where you get into the very bad throttling) [1] : https://software.intel.com/content/www/us/en/develop/blogs/c-states-and-p-states-are-very-different.html https://software.intel.com/content/www/us/en/develop/blogs/c...
  - jiggawatts 5y ago
    This might be the case for some artificial benchmark, but I have never seen such a case in practice. I have run many, many tests on real-world workloads, not artificial 100% loads of either a single core or all-cores, as you typically get in benchmarks. In all cases, disabling C-states made overall performance at least 20% better, and often 200% better (3x faster)! There's lots of reasons for this, and I could write pages and pages on how the jitter introduced by the sleeps is murder on TCP throughput, or how the frequency ramp up time is glacially slow, or how most servers are idle most of the time and only the cold-start latency matters to end-users. Then I could keep ranting about the waste of it all, and go on and on about how Californian greenie hippie morons tried to save the planet one database server at a time. Then finish up by showing that 10 Gbps Ethernet's latencies interact with the core sleep timings just so to make things as bad as possible. But instead of taking my word for it, you could just do your own tests on your own software, ideally in a production system involving n-tier networking instead of some isolated, synthetic benchmark. PS: By design, power management has no impact on anything running at 100% load for extended periods, so no synthetic benchmark will ever show any impact. That's precisely why very few people are aware of the issue...
hdjjhhvvhga 5y ago
> Provided their clients don't figure out what's going on As far as I can tell, very few people ask similar questions. Those who do, come to interesting conclusions: https://jan.rychter.com/enblog/cloud-server-cpu-performance-comparison-2019-12-12 https://jan.rychter.com/enblog/cloud-server-cpu-performance-...
- jiggawatts 5y ago
  It's so rare to see cloud benchmarks! I'm surprised this guy hasn't had a scary takedown letter from some lawyers yet... "There. I've said it. These [Xeon] processors are slow. Dog slow, in fact." My gimmick recently has been to upgrade Intel CPU VMs to AMD VMs. One benefit of the cloud is that this is literally just a menu selection. If it was on-prem, it would be a one-month argument to convince people that it's risk free, then a one-month argument with some beancounter, then a month of waiting while the pallets of kit ship in from Asia, then another two months of rack mounting and prepping for production. "They let them pack lots of slow cores onto a single CPU die, put that into a server, and then sell twice that number of cores as "vCPUs" to us." Azure also charges for licenses on a per vCPU basis as if* they were physical CPUs! This means that in effect they've doubled their Windows and SQL revenue stream.

jiggawatts 5y ago

> Provided their clients don't figure out what's going on I'm always pleasantly surprised when I see power management correctly configured at a client precisely because it is so rarely done correctly. Even if cloud vendors do fix this one thing, there's about a dozen more such issues where the suppliers' interests don't line up with the downstream consumers' interests. > Azure really don't offer any never-throttled instances? What about their dedicated hosts? I don't believe so. For example, their highest performing single node is the HBv3, with 2x AMD EPYC 7V13 processors for a total of 120 cores / 240 threads. On this page: https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/hpc/hbv3-series-overview https://docs.microsoft.com/en-us/azure/virtual-machines/work... There is a box that says: Nodes per Socket (NPS) = 2 L3 as NUMA = Disabled NUMA domains within VM OS = 4 C-states = Enabled The technical name for CPU power management is called "C-states". In other words, they've left it on the default (enabled) on a machine that they charge $40K/year for!

maayank 5y ago
> I'm always pleasantly surprised when I see power management correctly configured at a client precisely because it is so rarely done correctly. Let's say I have Linux instances in the cloud. What would be the steps to take for proper power management?
- jiggawatts 5y ago
  As far as I know, there are no steps you can take if you're running your instance in a virtual machine. Only the hypervisor host and/or the firmware can control the processor power and sleep states.
cwizou 5y ago
TL;DR: C-States improve performance. I'm not sure calling C-States throttling is completely accurate, at least not in the way that you imply. Modern CPUs have multiple mechanisms that can be called throttling in some way, but C-States are the least offensive and in modern CPUs actually increase performance. There's a good basic explanation here [1] but in short : - C-States is about idle cores, and powering them down. You can actually configure how deep you can power them down. Waking up does have a tiny cost in latency, yes, but it's mostly irrelevant (your core spends most of it's time waiting for data). - P-States are what I would call "ok" throttling, those happen when non-idle, modern cores can scale their frequency up and down based on how much workload they have. Again, in theory purely about power saving and close to 0 impact on performance. This was simple, and used to be just about power saving, disabling was ok. But things changed massively about 10+ years ago with the introductions of various Turbo mechanisms. Turbo is about looking at your CPU as a whole (and not per core), and maximising performance of the loaded cores by pushing their frequency, using the power budget that used to be allocated to the idle cores (that are now powered off). The way its done is by adding a few extra P-states that push the frequency a few hundred MHz. The more "C-State-down" cores you have, the higher the frequency of the working cores is pushed. Extreme example : you have a 28 cores CPUs running a single thread workload on one core only, with 27 idle cores. With C-States disabled, you lose Turbo and are stuck with the single core running at the "non-Turbo" frequency. With C-States enabled, because you idled the non used cores, your working core gets higher clock speeds and higher performances. (Things get more complex with AVX2 because those units have a large power consumption, and this is where you get into the very bad throttling) [1] : https://software.intel.com/content/www/us/en/develop/blogs/c-states-and-p-states-are-very-different.html https://software.intel.com/content/www/us/en/develop/blogs/c...
- jiggawatts 5y ago
  This might be the case for some artificial benchmark, but I have never seen such a case in practice. I have run many, many tests on real-world workloads, not artificial 100% loads of either a single core or all-cores, as you typically get in benchmarks. In all cases, disabling C-states made overall performance at least 20% better, and often 200% better (3x faster)! There's lots of reasons for this, and I could write pages and pages on how the jitter introduced by the sleeps is murder on TCP throughput, or how the frequency ramp up time is glacially slow, or how most servers are idle most of the time and only the cold-start latency matters to end-users. Then I could keep ranting about the waste of it all, and go on and on about how Californian greenie hippie morons tried to save the planet one database server at a time. Then finish up by showing that 10 Gbps Ethernet's latencies interact with the core sleep timings just so to make things as bad as possible. But instead of taking my word for it, you could just do your own tests on your own software, ideally in a production system involving n-tier networking instead of some isolated, synthetic benchmark. PS: By design, power management has no impact on anything running at 100% load for extended periods, so no synthetic benchmark will ever show any impact. That's precisely why very few people are aware of the issue...

hdjjhhvvhga 5y ago

> Provided their clients don't figure out what's going on As far as I can tell, very few people ask similar questions. Those who do, come to interesting conclusions: https://jan.rychter.com/enblog/cloud-server-cpu-performance-comparison-2019-12-12 https://jan.rychter.com/enblog/cloud-server-cpu-performance-...

jiggawatts 5y ago
It's so rare to see cloud benchmarks! I'm surprised this guy hasn't had a scary takedown letter from some lawyers yet... "There. I've said it. These [Xeon] processors are slow. Dog slow, in fact." My gimmick recently has been to upgrade Intel CPU VMs to AMD VMs. One benefit of the cloud is that this is literally just a menu selection. If it was on-prem, it would be a one-month argument to convince people that it's risk free, then a one-month argument with some beancounter, then a month of waiting while the pallets of kit ship in from Asia, then another two months of rack mounting and prepping for production. "They let them pack lots of slow cores onto a single CPU die, put that into a server, and then sell twice that number of cores as "vCPUs" to us." Azure also charges for licenses on a per vCPU basis as if* they were physical CPUs! This means that in effect they've doubled their Windows and SQL revenue stream.