3 ms·

> I'm not sure if "correcting" the numbers for 3 GHz is reasonable and reflects real-life performance It's not useful at all. It effectively measures IPC (inst

> I'm not sure if "correcting" the numbers for 3 GHz is reasonable and reflects real-life performance It's not useful at all. It effectively measures IPC (instructions per clock), which is just chip vendor bragging rights. Assuming that all the chips meet some baseline performance criteria: for datacenter and portable devices, the real benchmark would be "instructions per joule." For desktop devices "instructions per dollar" would be most relevant.

Octoth0rpe 4y ago
> For desktop devices "instructions per dollar" would be most relevant. For cloud customers as well
- cubefox 4y ago
  Which would probably include the cost of the chips in some way, not just electricity.
- zamalek 4y ago
  > For cloud customers as well Cloud costs are dominated by power delivery and cooling. Both of those are directly influenced by how much power the chip uses to achieve it's performance target. I guess it does indirectly influence dollar cost, but I was referring to MSRP of the chip. As a simple example: the per-chip cost of Graviton is probably enormous (if you factor R&D into the cost of a chip), but it's still cheaper for Amazon customers. Why? Power and cooling.
  - boulos 4y ago
    Disclosure: I used to work on GCE. I don't understand where these power and cooling mantras came from, but large-scale cloud providers have very low PUE (Google publishes historical data at https://www.google.com/about/datacenters/efficiency/ https://www.google.com/about/datacenters/efficiency/). That means you can take the basic power from a thread, add some for memory, and multiply by just a bit to get Watts. Take that and plug in your favorite $/kWh guess and get a price. Ignoring Graviton, which doesn't have published power data to my knowledge, you can look up the TDP for a bunch of chips and see that it's a a few Watts per thread [1]. Similar calculations can be done for RAM. You end up at 5ish Watts per thread with some RAM attached. Let's call it 10 all in with cooling or other stuff. Since 10W is 1% of a kW, we end up with .01 kWh per hour. The top hit for "us power commercial rates" [2] says that we should assume 7c per kWh or so. That means our instance with cooling and overheads, needs to include .01 x .07 => $.0007/hr of power and cooling costs. A single core w/ 4 GiB of memory on GCP at 3yr commitment rates (so we're focused on the long-term depreciation price) is .009815 + 4x.001316 => $.015/hr or about 20x as much as the power. tl;dr: Power costs add up, but they are not even close to dominating the costs of cloud pricing. [1] https://wccftech.com/amd-epyc-7h12-cpu-64-core-zen-2-280w-tdp-liquid-cooled/ https://wccftech.com/amd-epyc-7h12-cpu-64-core-zen-2-280w-td... [2] https://www.statista.com/statistics/190680/us-industrial-consumer-price-estimates-for-retail-electricity-since-1970 https://www.statista.com/statistics/190680/us-industrial-con...
    - jpgvm 4y ago
      Reaching those PUE numbers is pretty damn impressive though. I think the misconceptions come from enterprise DC environments with traditional hot/cold aisle designs, servers running way cooler than they need to be and peak power requirements leading to overly expensive power/cooling costs. PUE for these closer to 2.0 than GCE is to 1.0. If you have something like GCE where you can control for all of those, i.e run the DCs hot, eliminate transient peaks, source power cheaply and use super efficient cooling like evaporative or geothermal pumped water etc then yeah, it's a completely different ballgame. It's pretty safe to assume all the hyperscalers are doing all of these things too, they aren't stupid. :)
aylmao 4y ago
> It's not useful at all. It effectively measures IPC (instructions per clock), which is just chip vendor bragging rights. +1. Moreover the author then seems to conclude from this benchmark: > Overall, these numbers suggest that the Qualcomm processor is competitive. This is an odd conclusion to draw from this test and these numbers, given how little this benchmark tests (just string operations). Does this benchmark want to test raw CPU power? Then why "normalize" to 3GHz? Does it want to test CPU capabilities? If so why use such a "narrow" test? IMO this benchmarking does for a good data-point, but far from enough to draw much of a conclusion from.
xoa 4y ago
>For desktop devices "instructions per dollar" would be most relevant. Well, it's one measure, but scaling potential and absolute performance limit matters as well. In use cases where a desktop is performing work to help one or more humans, the cost of salary and other support may utterly drown out even a very expensive extremely high power system. Ie., a 1200W workstation being used at maximum power 10 hours a day 365 days a year (an absurd utilization ratio) at $0.15/kWh (average US electricity cost) would still only be around $650/year. If it boosted the productivity of a typical tech worker even 1% it'd pay for itself no problem. It's easy to lose sight sometimes of how historically incredible the bang for the buck is in tech. I think it's important to remember because some designs that do incredibly well at small sizes run into challenges scaling. Like with CPUs, the nature of silicon fabrication makes it ever more difficult to grow a monolithic die. Switching to a chiplet-based design does mean absolute efficiency and minimum power challenges amongst others, but dramatically improves scalability at the high end. It's a sort of infrastructure tradeoff. Apple for example has done incredibly well with its big silicon SoCs from handhelds to portable Macs, but it's been struggling to do a Mac Pro or even updated Studio. It's not clear that they physically can do something on the level of a modern Epyc chip anymore than Intel could with monolithic Xeons. So normalized performance/watt and performance/$ and so on do matter, but absolute final density and scale up are another part of the matrix for certain use cases too.
- jnsaff2 4y ago
  Your tech worker only gets about 4.1 days of not working? Wow. That aside, I think it would be more important to look at the carbon emissions impact of code/cpu's. That one person parsing urls might save few hundred dollars of not optimising their code but if it is being used billions of times a day the energy (and therefore emissions) impact can be huge.
Aeolun 4y ago
Maybe for you? If I’m looking at a new CPU for my desktop the only thing really relevant is how many instructions it can crank out per second. I’m not trying to optimize for cost or energy efficiency.
- gpderetta 4y ago
  Instructions per second is an useful performance metric. Instructions per clock isn't.
adfghuibnuio 4y ago
It's pretty amusing to me how the Megahertz Myth has been inverted nowadays. It used to be everyone cared about the fastest clock speed and irrationally discounted IPC. Now people care about IPC and irrationally discount clock speed. Pretty weird, in my opinion.
- saagarjha 4y ago
  Clock speed is comparatively easier to boost.
foobiekr 4y ago
Your typical DSP beats your typical CPU on power and instructions/sec/dollar by a pretty wide margin.
- gpderetta 4y ago
  You have to measure effective (not theoretical peak) instructions on a specific workload. DSPs will beat CPUs on DSP workloads, but, as expected, utterly fail for any general purpose workload.
  - foobiekr 4y ago
    Clearly so.

Octoth0rpe 4y ago

> For desktop devices "instructions per dollar" would be most relevant. For cloud customers as well

cubefox 4y ago
Which would probably include the cost of the chips in some way, not just electricity.
zamalek 4y ago
> For cloud customers as well Cloud costs are dominated by power delivery and cooling. Both of those are directly influenced by how much power the chip uses to achieve it's performance target. I guess it does indirectly influence dollar cost, but I was referring to MSRP of the chip. As a simple example: the per-chip cost of Graviton is probably enormous (if you factor R&D into the cost of a chip), but it's still cheaper for Amazon customers. Why? Power and cooling.
- boulos 4y ago
  Disclosure: I used to work on GCE. I don't understand where these power and cooling mantras came from, but large-scale cloud providers have very low PUE (Google publishes historical data at https://www.google.com/about/datacenters/efficiency/ https://www.google.com/about/datacenters/efficiency/). That means you can take the basic power from a thread, add some for memory, and multiply by just a bit to get Watts. Take that and plug in your favorite $/kWh guess and get a price. Ignoring Graviton, which doesn't have published power data to my knowledge, you can look up the TDP for a bunch of chips and see that it's a a few Watts per thread [1]. Similar calculations can be done for RAM. You end up at 5ish Watts per thread with some RAM attached. Let's call it 10 all in with cooling or other stuff. Since 10W is 1% of a kW, we end up with .01 kWh per hour. The top hit for "us power commercial rates" [2] says that we should assume 7c per kWh or so. That means our instance with cooling and overheads, needs to include .01 x .07 => $.0007/hr of power and cooling costs. A single core w/ 4 GiB of memory on GCP at 3yr commitment rates (so we're focused on the long-term depreciation price) is .009815 + 4x.001316 => $.015/hr or about 20x as much as the power. tl;dr: Power costs add up, but they are not even close to dominating the costs of cloud pricing. [1] https://wccftech.com/amd-epyc-7h12-cpu-64-core-zen-2-280w-tdp-liquid-cooled/ https://wccftech.com/amd-epyc-7h12-cpu-64-core-zen-2-280w-td... [2] https://www.statista.com/statistics/190680/us-industrial-consumer-price-estimates-for-retail-electricity-since-1970 https://www.statista.com/statistics/190680/us-industrial-con...
  - jpgvm 4y ago
    Reaching those PUE numbers is pretty damn impressive though. I think the misconceptions come from enterprise DC environments with traditional hot/cold aisle designs, servers running way cooler than they need to be and peak power requirements leading to overly expensive power/cooling costs. PUE for these closer to 2.0 than GCE is to 1.0. If you have something like GCE where you can control for all of those, i.e run the DCs hot, eliminate transient peaks, source power cheaply and use super efficient cooling like evaporative or geothermal pumped water etc then yeah, it's a completely different ballgame. It's pretty safe to assume all the hyperscalers are doing all of these things too, they aren't stupid. :)

aylmao 4y ago

> It's not useful at all. It effectively measures IPC (instructions per clock), which is just chip vendor bragging rights. +1. Moreover the author then seems to conclude from this benchmark: > Overall, these numbers suggest that the Qualcomm processor is competitive. This is an odd conclusion to draw from this test and these numbers, given how little this benchmark tests (just string operations). Does this benchmark want to test raw CPU power? Then why "normalize" to 3GHz? Does it want to test CPU capabilities? If so why use such a "narrow" test? IMO this benchmarking does for a good data-point, but far from enough to draw much of a conclusion from.

xoa 4y ago

>For desktop devices "instructions per dollar" would be most relevant. Well, it's one measure, but scaling potential and absolute performance limit matters as well. In use cases where a desktop is performing work to help one or more humans, the cost of salary and other support may utterly drown out even a very expensive extremely high power system. Ie., a 1200W workstation being used at maximum power 10 hours a day 365 days a year (an absurd utilization ratio) at $0.15/kWh (average US electricity cost) would still only be around $650/year. If it boosted the productivity of a typical tech worker even 1% it'd pay for itself no problem. It's easy to lose sight sometimes of how historically incredible the bang for the buck is in tech. I think it's important to remember because some designs that do incredibly well at small sizes run into challenges scaling. Like with CPUs, the nature of silicon fabrication makes it ever more difficult to grow a monolithic die. Switching to a chiplet-based design does mean absolute efficiency and minimum power challenges amongst others, but dramatically improves scalability at the high end. It's a sort of infrastructure tradeoff. Apple for example has done incredibly well with its big silicon SoCs from handhelds to portable Macs, but it's been struggling to do a Mac Pro or even updated Studio. It's not clear that they physically can do something on the level of a modern Epyc chip anymore than Intel could with monolithic Xeons. So normalized performance/watt and performance/$ and so on do matter, but absolute final density and scale up are another part of the matrix for certain use cases too.

jnsaff2 4y ago
Your tech worker only gets about 4.1 days of not working? Wow. That aside, I think it would be more important to look at the carbon emissions impact of code/cpu's. That one person parsing urls might save few hundred dollars of not optimising their code but if it is being used billions of times a day the energy (and therefore emissions) impact can be huge.

Aeolun 4y ago

Maybe for you? If I’m looking at a new CPU for my desktop the only thing really relevant is how many instructions it can crank out per second. I’m not trying to optimize for cost or energy efficiency.

gpderetta 4y ago
Instructions per second is an useful performance metric. Instructions per clock isn't.

adfghuibnuio 4y ago

It's pretty amusing to me how the Megahertz Myth has been inverted nowadays. It used to be everyone cared about the fastest clock speed and irrationally discounted IPC. Now people care about IPC and irrationally discount clock speed. Pretty weird, in my opinion.

saagarjha 4y ago
Clock speed is comparatively easier to boost.

foobiekr 4y ago

Your typical DSP beats your typical CPU on power and instructions/sec/dollar by a pretty wide margin.

gpderetta 4y ago
You have to measure effective (not theoretical peak) instructions on a specific workload. DSPs will beat CPUs on DSP workloads, but, as expected, utterly fail for any general purpose workload.
- foobiekr 4y ago
  Clearly so.