5 ms·

A comparison with x86_64 CPUs (e.g. those seen in comparable MacBooks and AWS machines) would be useful. Also, I'm not sure if "correcting" the numbers for 3 G

A comparison with x86_64 CPUs (e.g. those seen in comparable MacBooks and AWS machines) would be useful. Also, I'm not sure if "correcting" the numbers for 3 GHz is reasonable and reflects real-life performance. Perhaps some throttling could be applied to test the CPUs using a common frequency?

deltaci 4y ago
It's a benchmark of GitHub Actions(Azure) vs a really old Macbook Pro 15, not exactly what you are looking for, but it tells the vibe already. https://buildjet.com/for-github-actions/blog/a-performance-review-of-github-actions-the-cost-of-slow-hardware https://buildjet.com/for-github-actions/blog/a-performance-r...
- 015a 4y ago
  This is a big, general problem with CI providers I don't hear talked about enough: because they charge per-minute, they are actively incentivized to run on old hardware, slowing builds and milking more from customers in the process. Doubly-so when your CI is hosted by a major cloud provider who would otherwise have to scrap these old machines. I wish this were only a theoretical concern, a theoretical incentive, but its not. Github Actions is slow, and Gitlab suffers from a similar problem; their hosted SaaS runners are on GCP n1-standard-1 machines. The oldest machine type in GCP's fleet, the n1-standard-1 is powered by a variety of dusty, old CPUs Google Cloud has no other use for, from Sandy Bridge to Skylake. That's a 12 year old CPU.
  - AnthonyMouse 4y ago
    There are workloads where newer CPUs are dramatically faster (e.g. AVX-512), but in general the difference isn't huge. Most of what the newer CPUs get you is more cores and higher power efficiency, which you don't care about when you're paying per-vCPU. Which vCPU is faster, a ten year old Xeon E5-2643 v2 at 3.5GHz or a two year old Xeon Platinum 8352V at 2.1GHz? It depends on the workload. Which has more memory bandwidth per core? But the cloud provider prefers the latter because it has 500% more cores for 50% more power. Which is why the latter still goes for >$2000 and the former is <$15.
    - 015a 4y ago
      > Which vCPU is faster, a ten year old Xeon E5-2643 v2 at 3.5GHz or a two year old Xeon Platinum 8352V at 2.1GHz? It depends on the workload. It really does not depend on the workload, when those workloads we're talking about are by-and-large bounded to 1vCPU or less (CI jobs, serverless functions, etc). Ice Lake cores are substantially faster than Ivy Bridge; the 8352V will be faster in practically any workload we're talking about. However, I do agree with this take, if we're talking about, say, lambda functions. The reason being that the vast majority of workloads built on lambda functions are bounded by IO, not compute; so newer core designs won't result in a meaningful improvement in function execution. Put another way: Is a function executing in 75ms instead of 80ms worth paying 30% more? (I made these numbers up, but its the illustration that matters). CI is a different story. CI runs are only bound by IO for the smallest of projects; downloading that 800mb node:18 base docker image takes some time, but it can very easily and quickly be dwarfed by all the things that happen afterward. This is not an uncontroversial opinion; "the CI is slow" is such a meme of a problem at engineering companies nowadays that you'd think more people would have the sense to look at the common denominator (the CI hosts suck) and not blame themselves (though, often there's blame to go around). We've got a project that can build locally, M2 Pro, docker pull and push included, in something like 40 seconds; the CI takes 4 minutes. Its the crusty CPUs; its slow networking; its the "step 1 is finished, wait 10 seconds for the orchestrator to realize it and start step 2". And I think we, the community, need to be more vocal about this when speaking on platforms that charge by the minute. They are clearly incentivized to leave it shitty. It should even surface in discussions about, for example, the markup of lambda versus EC2. A 4096mb lambda function would cost $172/mo if ran 24/7, back-to-back. A comparable c6i-large: $62/mo; a third the price. That's bad enough on the surface, and we need to be cognizant that its even worse than it initially appears because Amazon runs Lambda on whatever they have collecting dust in the closet, and people still report getting Ivy Bridge and Haswell cores sometimes, in 2023; and the better comparison is probably a t2-medium @ $33/mo; a 5-6x markup. This isn't new information; lambda is crazy expensive; blah blah blah; but I don't hear that dimension brought up enough. Calling back to my previous point: Is a function executing in 75ms instead of 80ms worth paying 30% more? Well, we're already paying 550% more; the fact that it doesn't execute in 75ms by default is abhorrent. Put another way: if Lambda, and other serverless systems like it such as hosted CI runners, enables cloud providers to keep old hardware around far longer than performance improvements say it should be; the markup should not be 500%. We're doing Amazon a favor by using Lambda.
- willcipriano 4y ago
  Sometimes when I run a lot of builds in a short period of time I feel like I get demoted to the slower boxes.
stingraycharles 4y ago
In my totally unscientific (but consistent) benchmarks for our CI build servers, m6g.8xlarge compile our c++ codebase in about 9.5 minutes, where m6a.8xlarge takes about 11 minutes. The price difference is about 20% as well iirc, so it’s generally a good deal. Of course the types of optimisations that a compiler may (or may not) do on aarch64 vs x86_64 are completely different and may explain the difference (we actually compile with -march=haswell for x86_64), but generally Graviton seems like a really good deal.
- foota 4y ago
  You're probably leaving a lot of performance on the floor if you're building for haskell and running on skylakeish or newer. Edit: yes, haswell:-)
  - nickpeterson 4y ago
    *haswell, in case the Haskell people come after you.
    - paulddraper 4y ago
      Ah, that makes more sense.
    - speed_spread 4y ago
      "don't poke the endofunctor"
  - stingraycharles 4y ago
    Yeah we know, it’s just that we ship these binaries to customers and need a reasonably old architecture that we know “almost everyone” uses.
- Aeolun 4y ago
  Why not use the c type instances? Is the compilation memory bound? I’d expect it to be the cpu that’s the limiting factor.
zamalek 4y ago
> I'm not sure if "correcting" the numbers for 3 GHz is reasonable and reflects real-life performance It's not useful at all. It effectively measures IPC (instructions per clock), which is just chip vendor bragging rights. Assuming that all the chips meet some baseline performance criteria: for datacenter and portable devices, the real benchmark would be "instructions per joule." For desktop devices "instructions per dollar" would be most relevant.
- Octoth0rpe 4y ago
  > For desktop devices "instructions per dollar" would be most relevant. For cloud customers as well
  - cubefox 4y ago
    Which would probably include the cost of the chips in some way, not just electricity.
  - zamalek 4y ago
    > For cloud customers as well Cloud costs are dominated by power delivery and cooling. Both of those are directly influenced by how much power the chip uses to achieve it's performance target. I guess it does indirectly influence dollar cost, but I was referring to MSRP of the chip. As a simple example: the per-chip cost of Graviton is probably enormous (if you factor R&D into the cost of a chip), but it's still cheaper for Amazon customers. Why? Power and cooling.
    - boulos 4y ago
      Disclosure: I used to work on GCE. I don't understand where these power and cooling mantras came from, but large-scale cloud providers have very low PUE (Google publishes historical data at https://www.google.com/about/datacenters/efficiency/ https://www.google.com/about/datacenters/efficiency/). That means you can take the basic power from a thread, add some for memory, and multiply by just a bit to get Watts. Take that and plug in your favorite $/kWh guess and get a price. Ignoring Graviton, which doesn't have published power data to my knowledge, you can look up the TDP for a bunch of chips and see that it's a a few Watts per thread [1]. Similar calculations can be done for RAM. You end up at 5ish Watts per thread with some RAM attached. Let's call it 10 all in with cooling or other stuff. Since 10W is 1% of a kW, we end up with .01 kWh per hour. The top hit for "us power commercial rates" [2] says that we should assume 7c per kWh or so. That means our instance with cooling and overheads, needs to include .01 x .07 => $.0007/hr of power and cooling costs. A single core w/ 4 GiB of memory on GCP at 3yr commitment rates (so we're focused on the long-term depreciation price) is .009815 + 4x.001316 => $.015/hr or about 20x as much as the power. tl;dr: Power costs add up, but they are not even close to dominating the costs of cloud pricing. [1] https://wccftech.com/amd-epyc-7h12-cpu-64-core-zen-2-280w-tdp-liquid-cooled/ https://wccftech.com/amd-epyc-7h12-cpu-64-core-zen-2-280w-td... [2] https://www.statista.com/statistics/190680/us-industrial-consumer-price-estimates-for-retail-electricity-since-1970 https://www.statista.com/statistics/190680/us-industrial-con...
pr0zac 4y ago
It feels like a pretty useless benchmark as far as translating to real world performance but for the heck of it I did ten benchmarks on my three year old Thinkpad with a 4.5Ghz Intel i7-9750H running Ubuntu and got a best of ns/url=225.63 and worst of ns/url=275.118 whatever those scores mean.
nspattak 4y ago
The author very clearly and explicitely says in the post that this is "purely to help us think".
tlamponi 4y ago
On my (not idle) i7-12700K with DDR5 4800 memory Workstation, using Proxmox VE 7.4 (based on Debian 11) with our 6.2 based Linux kernel, I get something between 139.44 ns/url and 140.66 ns/url (ran it about 10 times in a row), so 1.35x faster than the M2.
- heredoc 4y ago
  I constantly get between 154 and 156 ns/url when running on an M2 Air 2022 while using asahi linux instead of MacOs. 22 Celsius according to smartmontools :)
  - tlamponi 4y ago
    So seems a good part of the speed-up comes from using the Linux kernel, which tbh. doesn't really surprise me. It's hard to match the engineering amount and quality getting poured into it (well at least the core stuff, some niche driver will be a mess like everywhere). The remainder then comes from modern x86_64/amd64 CPU combined with DDR5 beating the M2, well at least in this micro benchmark.
- StillBored 4y ago
  One of the comments on the article notes a 7950x doing "time/url=109ns" so I tossed it on my 5950x and it reports "ns/url=148.994" which is about right from what I understand (zen4 gets a nice ~30% bump on a number of Linux benchmarks over zen3), here its 35% with only a 16% clock advantage. Which is about the diff between the M2 and my old 5950X in a generation older process/etc. So, yah current M2 slower than previous gen Intel and AMD both in this benchmark.
eek2121 4y ago
perf/watt/density would be the best way to measure. Frequency should NOT be normalized because x86 chips are designed to be high frequency parts. That is why measuring IPC is meaningless. A cutting edge ARM chip can possibly beat a modern x86 chip in terms of IPC, but the x86 chip will almost certainly scale to higher frequencies. Certain models of EPYC Genoa (Zen 4) for example, boost to 4GHz, and both Intel and AMD have chips that boost to 5.7+ GHz. Density is important because if you can only have a max of, for example, 20 cores for an ARM solution, but 96 cores in the case of EPYC Genoa, Genoa is going to win out for any multi-core workload.

deltaci 4y ago

It's a benchmark of GitHub Actions(Azure) vs a really old Macbook Pro 15, not exactly what you are looking for, but it tells the vibe already. https://buildjet.com/for-github-actions/blog/a-performance-review-of-github-actions-the-cost-of-slow-hardware https://buildjet.com/for-github-actions/blog/a-performance-r...

015a 4y ago
This is a big, general problem with CI providers I don't hear talked about enough: because they charge per-minute, they are actively incentivized to run on old hardware, slowing builds and milking more from customers in the process. Doubly-so when your CI is hosted by a major cloud provider who would otherwise have to scrap these old machines. I wish this were only a theoretical concern, a theoretical incentive, but its not. Github Actions is slow, and Gitlab suffers from a similar problem; their hosted SaaS runners are on GCP n1-standard-1 machines. The oldest machine type in GCP's fleet, the n1-standard-1 is powered by a variety of dusty, old CPUs Google Cloud has no other use for, from Sandy Bridge to Skylake. That's a 12 year old CPU.
- AnthonyMouse 4y ago
  There are workloads where newer CPUs are dramatically faster (e.g. AVX-512), but in general the difference isn't huge. Most of what the newer CPUs get you is more cores and higher power efficiency, which you don't care about when you're paying per-vCPU. Which vCPU is faster, a ten year old Xeon E5-2643 v2 at 3.5GHz or a two year old Xeon Platinum 8352V at 2.1GHz? It depends on the workload. Which has more memory bandwidth per core? But the cloud provider prefers the latter because it has 500% more cores for 50% more power. Which is why the latter still goes for >$2000 and the former is <$15.
  - 015a 4y ago
    > Which vCPU is faster, a ten year old Xeon E5-2643 v2 at 3.5GHz or a two year old Xeon Platinum 8352V at 2.1GHz? It depends on the workload. It really does not depend on the workload, when those workloads we're talking about are by-and-large bounded to 1vCPU or less (CI jobs, serverless functions, etc). Ice Lake cores are substantially faster than Ivy Bridge; the 8352V will be faster in practically any workload we're talking about. However, I do agree with this take, if we're talking about, say, lambda functions. The reason being that the vast majority of workloads built on lambda functions are bounded by IO, not compute; so newer core designs won't result in a meaningful improvement in function execution. Put another way: Is a function executing in 75ms instead of 80ms worth paying 30% more? (I made these numbers up, but its the illustration that matters). CI is a different story. CI runs are only bound by IO for the smallest of projects; downloading that 800mb node:18 base docker image takes some time, but it can very easily and quickly be dwarfed by all the things that happen afterward. This is not an uncontroversial opinion; "the CI is slow" is such a meme of a problem at engineering companies nowadays that you'd think more people would have the sense to look at the common denominator (the CI hosts suck) and not blame themselves (though, often there's blame to go around). We've got a project that can build locally, M2 Pro, docker pull and push included, in something like 40 seconds; the CI takes 4 minutes. Its the crusty CPUs; its slow networking; its the "step 1 is finished, wait 10 seconds for the orchestrator to realize it and start step 2". And I think we, the community, need to be more vocal about this when speaking on platforms that charge by the minute. They are clearly incentivized to leave it shitty. It should even surface in discussions about, for example, the markup of lambda versus EC2. A 4096mb lambda function would cost $172/mo if ran 24/7, back-to-back. A comparable c6i-large: $62/mo; a third the price. That's bad enough on the surface, and we need to be cognizant that its even worse than it initially appears because Amazon runs Lambda on whatever they have collecting dust in the closet, and people still report getting Ivy Bridge and Haswell cores sometimes, in 2023; and the better comparison is probably a t2-medium @ $33/mo; a 5-6x markup. This isn't new information; lambda is crazy expensive; blah blah blah; but I don't hear that dimension brought up enough. Calling back to my previous point: Is a function executing in 75ms instead of 80ms worth paying 30% more? Well, we're already paying 550% more; the fact that it doesn't execute in 75ms by default is abhorrent. Put another way: if Lambda, and other serverless systems like it such as hosted CI runners, enables cloud providers to keep old hardware around far longer than performance improvements say it should be; the markup should not be 500%. We're doing Amazon a favor by using Lambda.
willcipriano 4y ago
Sometimes when I run a lot of builds in a short period of time I feel like I get demoted to the slower boxes.

stingraycharles 4y ago

In my totally unscientific (but consistent) benchmarks for our CI build servers, m6g.8xlarge compile our c++ codebase in about 9.5 minutes, where m6a.8xlarge takes about 11 minutes. The price difference is about 20% as well iirc, so it’s generally a good deal. Of course the types of optimisations that a compiler may (or may not) do on aarch64 vs x86_64 are completely different and may explain the difference (we actually compile with -march=haswell for x86_64), but generally Graviton seems like a really good deal.

foota 4y ago
You're probably leaving a lot of performance on the floor if you're building for haskell and running on skylakeish or newer. Edit: yes, haswell:-)
- nickpeterson 4y ago
  *haswell, in case the Haskell people come after you.
  - paulddraper 4y ago
    Ah, that makes more sense.
  - speed_spread 4y ago
    "don't poke the endofunctor"
- stingraycharles 4y ago
  Yeah we know, it’s just that we ship these binaries to customers and need a reasonably old architecture that we know “almost everyone” uses.
Aeolun 4y ago
Why not use the c type instances? Is the compilation memory bound? I’d expect it to be the cpu that’s the limiting factor.

zamalek 4y ago

> I'm not sure if "correcting" the numbers for 3 GHz is reasonable and reflects real-life performance It's not useful at all. It effectively measures IPC (instructions per clock), which is just chip vendor bragging rights. Assuming that all the chips meet some baseline performance criteria: for datacenter and portable devices, the real benchmark would be "instructions per joule." For desktop devices "instructions per dollar" would be most relevant.

Octoth0rpe 4y ago
> For desktop devices "instructions per dollar" would be most relevant. For cloud customers as well
- cubefox 4y ago
  Which would probably include the cost of the chips in some way, not just electricity.
- zamalek 4y ago
  > For cloud customers as well Cloud costs are dominated by power delivery and cooling. Both of those are directly influenced by how much power the chip uses to achieve it's performance target. I guess it does indirectly influence dollar cost, but I was referring to MSRP of the chip. As a simple example: the per-chip cost of Graviton is probably enormous (if you factor R&D into the cost of a chip), but it's still cheaper for Amazon customers. Why? Power and cooling.
  - boulos 4y ago
    Disclosure: I used to work on GCE. I don't understand where these power and cooling mantras came from, but large-scale cloud providers have very low PUE (Google publishes historical data at https://www.google.com/about/datacenters/efficiency/ https://www.google.com/about/datacenters/efficiency/). That means you can take the basic power from a thread, add some for memory, and multiply by just a bit to get Watts. Take that and plug in your favorite $/kWh guess and get a price. Ignoring Graviton, which doesn't have published power data to my knowledge, you can look up the TDP for a bunch of chips and see that it's a a few Watts per thread [1]. Similar calculations can be done for RAM. You end up at 5ish Watts per thread with some RAM attached. Let's call it 10 all in with cooling or other stuff. Since 10W is 1% of a kW, we end up with .01 kWh per hour. The top hit for "us power commercial rates" [2] says that we should assume 7c per kWh or so. That means our instance with cooling and overheads, needs to include .01 x .07 => $.0007/hr of power and cooling costs. A single core w/ 4 GiB of memory on GCP at 3yr commitment rates (so we're focused on the long-term depreciation price) is .009815 + 4x.001316 => $.015/hr or about 20x as much as the power. tl;dr: Power costs add up, but they are not even close to dominating the costs of cloud pricing. [1] https://wccftech.com/amd-epyc-7h12-cpu-64-core-zen-2-280w-tdp-liquid-cooled/ https://wccftech.com/amd-epyc-7h12-cpu-64-core-zen-2-280w-td... [2] https://www.statista.com/statistics/190680/us-industrial-consumer-price-estimates-for-retail-electricity-since-1970 https://www.statista.com/statistics/190680/us-industrial-con...

pr0zac 4y ago

It feels like a pretty useless benchmark as far as translating to real world performance but for the heck of it I did ten benchmarks on my three year old Thinkpad with a 4.5Ghz Intel i7-9750H running Ubuntu and got a best of ns/url=225.63 and worst of ns/url=275.118 whatever those scores mean.

nspattak 4y ago

The author very clearly and explicitely says in the post that this is "purely to help us think".

tlamponi 4y ago

On my (not idle) i7-12700K with DDR5 4800 memory Workstation, using Proxmox VE 7.4 (based on Debian 11) with our 6.2 based Linux kernel, I get something between 139.44 ns/url and 140.66 ns/url (ran it about 10 times in a row), so 1.35x faster than the M2.

heredoc 4y ago
I constantly get between 154 and 156 ns/url when running on an M2 Air 2022 while using asahi linux instead of MacOs. 22 Celsius according to smartmontools :)
- tlamponi 4y ago
  So seems a good part of the speed-up comes from using the Linux kernel, which tbh. doesn't really surprise me. It's hard to match the engineering amount and quality getting poured into it (well at least the core stuff, some niche driver will be a mess like everywhere). The remainder then comes from modern x86_64/amd64 CPU combined with DDR5 beating the M2, well at least in this micro benchmark.
StillBored 4y ago
One of the comments on the article notes a 7950x doing "time/url=109ns" so I tossed it on my 5950x and it reports "ns/url=148.994" which is about right from what I understand (zen4 gets a nice ~30% bump on a number of Linux benchmarks over zen3), here its 35% with only a 16% clock advantage. Which is about the diff between the M2 and my old 5950X in a generation older process/etc. So, yah current M2 slower than previous gen Intel and AMD both in this benchmark.

eek2121 4y ago

perf/watt/density would be the best way to measure. Frequency should NOT be normalized because x86 chips are designed to be high frequency parts. That is why measuring IPC is meaningless. A cutting edge ARM chip can possibly beat a modern x86 chip in terms of IPC, but the x86 chip will almost certainly scale to higher frequencies. Certain models of EPYC Genoa (Zen 4) for example, boost to 4GHz, and both Intel and AMD have chips that boost to 5.7+ GHz. Density is important because if you can only have a max of, for example, 20 cores for an ARM solution, but 96 cores in the case of EPYC Genoa, Genoa is going to win out for any multi-core workload.