10 ms·

Evaluating Graviton 2 for data-intensive applications: Arm vs. Intel comparison

62 points by Aissen 5y ago

xhkkffbf 5y ago
Does anyone have any insight into the Graviton 3? I'm fascinated by their single precision floating point which can be useful for some ML models.
- BeeOnRope 5y ago
  It looks like a strong chip. They are pretty much jumping one or two x86 generations with every release, sort of like what Apple did for many generations in a row starting with the A6 or whatever. One difference here is that Amazon isn't actually designing the cores: they are using Arm reference designs: in this case the N1 Neoverse, and building around that.
  - sydthrowaway 5y ago
    Amazon bought Annapurna though - why aren’t they building from scratch?
    - BeeOnRope 5y ago
      They may do it at some point, but perhaps the path of least resistance so far is to use the existing core designs, which are not bad at all and getting better and spending their effort on the uncore and offcore components which don't have off-the-shelf designs just sitting there and which are very important for server parts?
- ksec 5y ago
  Oh Gosh we are in April already and Graviton 3 was announced 4 - 5 months ago? I was hoping we have pricing and more performance figure by now.
bhouston 5y ago
Interesting to see that it has higher throughput per $. That is the most important fact.
- BeeOnRope 5y ago
  Yeah, these instance types look strong and it seems that Amazon is also innovating in the local storage space as well as I don't find many of the usual gotchas with SSDs there (long GC pauses, etc). We may be moving to a future where cloud hardware stops being "just OK" and a low maintenance version of what you could build yourself, but actually offering unobtainable-by-mere-mortals hardware and firmware (we are already seeing this for e.g. with quantum computing options in the cloud and even arguably GPUs and ML accelerators).
  - ceeplusplus 5y ago
    It is still vastly cheaper to buy your own GPUs and ML accelerators than to rent from cloud providers for any reasonable level of utilization (i.e. not a hobby).
    - BeeOnRope 5y ago
      My point was that there are accelerators in the cloud which are not available to purchase by the general public and for a time the same applied also to GPUs in a practical sense (i.e., GPUs were being sold in principle but were out of stock for months on end). Depending on your individual time value of money, it can make sense to rent even a high rate so you can access this hardware now.
    - ip26 5y ago
      I'm excited on behalf of the scientists I know for cloud HPC. I see a lot of "develop program locally on tiny subset of data" followed by "how the hell am I ever going to run the full model for the final run on all the data!?" A couple hours machine time leased on a behemoth is really the ticket, and not really that expensive.
      - 5y ago
BeeOnRope 5y ago
I wrote this, happy for any feedback or to answer any question about the methodology or otherwise. One thing note is that the headline difference between these instances types is the CPU architecture, the value offered by these instance types is as much about how many IOPS and how much SSD space EC2 chooses to offer for each family, especially for this IO-bound benchmark. It is entirely plausible that EC2 will offer more hardware bang-for-buck on Graviton instances in an effort to encourage adoption so we can't necessarily draw a strong conclusion about the future of the Arm vs x86 battle on the server CPU front since Amazon is not an Arms-length entity here (pun intended).
- Uehreka 5y ago
  One bit of feedback, it’s a good idea to put “higher is better” or “lower is better” on graph axes.
  - BeeOnRope 5y ago
    Thanks, this is great feedback that I'll implement next time. You are right that it is not always obvious especially when talking about both throughput (higher is better) and latency (lower is better) in the same post.
    - bpicolo 5y ago
      I might be missing the context, but I think the units on the graphs might be off? Currently "Bytes per second". Guessing there should be a multiplier on that?
      - BeeOnRope 5y ago
        Indeed, the multiplier is 1e9. It was there in the original matplotlib charts, but was dropped in the "chart beautification" process, probably due to the way mpl likes to hide this detail off in a corner somewhere.
CyanLite2 5y ago
cliff notes? Site seems to have been hugged to death
- pzarex 5y ago
  Link works for me... TL;DR, Arm instances give better bang for the buck...
- speed_spread 5y ago
  Probably hosted on Graviton instances
sharpy 5y ago
Recently worked on migrating a number of services (largely CPU bound) to Graviton 2. It was a bit of hit and mess. For some services, it worked great. Very negligible hit to performance, and since equivalent Graviton2 instances cost 20% less, it was a no brainer. Others needed a little bit of rewrite. There were also a couple that just didn't see good performance on ARM (very branchy code)
- BeeOnRope 5y ago
  Interesting. Other than branchy-ness, was there anything else you think correlated to poorer performance on Graviton 2.
  - ceeplusplus 5y ago
    Branchy-ness usually also correlates with cache friendliness (e.g. branchy code is usually a server app calling lots of different functions, with lots of virtual function calls), and Graviton2 has an abnormally small L3 cache relative to its x86 competitors. If I had to hazard a guess I'd say heavily vectorized code will also perform worse especially compared to Intel instances where you have AVX-512.
    - winrid 5y ago
      Didn't read into this much yet but the smaller L3 cache makes me sad. Benchmarking will show true colors but hoping to move some DB clusters to Gravaton2.
redwood 5y ago
Got to feel bad for Intel
phonon 5y ago
You're comparing 4th generation storage to third generation. Unfortunately, the I4i instances are not publically available, despite being announced 4 months ago :-( https://press.aboutamazon.com/news-releases/news-release-details/aws-announces-three-new-amazon-ec2-instances-powered-aws https://press.aboutamazon.com/news-releases/news-release-det... https://aws.amazon.com/ec2/instance-types/i4i/ https://aws.amazon.com/ec2/instance-types/i4i/

62 points by Aissen 5y ago

Does anyone have any insight into the Graviton 3? I'm fascinated by their single precision floating point which can be useful for some ML models.

BeeOnRope 5y ago
It looks like a strong chip. They are pretty much jumping one or two x86 generations with every release, sort of like what Apple did for many generations in a row starting with the A6 or whatever. One difference here is that Amazon isn't actually designing the cores: they are using Arm reference designs: in this case the N1 Neoverse, and building around that.
- sydthrowaway 5y ago
  Amazon bought Annapurna though - why aren’t they building from scratch?
  - BeeOnRope 5y ago
    They may do it at some point, but perhaps the path of least resistance so far is to use the existing core designs, which are not bad at all and getting better and spending their effort on the uncore and offcore components which don't have off-the-shelf designs just sitting there and which are very important for server parts?
ksec 5y ago
Oh Gosh we are in April already and Graviton 3 was announced 4 - 5 months ago? I was hoping we have pricing and more performance figure by now.

bhouston 5y ago

Interesting to see that it has higher throughput per $. That is the most important fact.

BeeOnRope 5y ago
Yeah, these instance types look strong and it seems that Amazon is also innovating in the local storage space as well as I don't find many of the usual gotchas with SSDs there (long GC pauses, etc). We may be moving to a future where cloud hardware stops being "just OK" and a low maintenance version of what you could build yourself, but actually offering unobtainable-by-mere-mortals hardware and firmware (we are already seeing this for e.g. with quantum computing options in the cloud and even arguably GPUs and ML accelerators).
- ceeplusplus 5y ago
  It is still vastly cheaper to buy your own GPUs and ML accelerators than to rent from cloud providers for any reasonable level of utilization (i.e. not a hobby).
  - BeeOnRope 5y ago
    My point was that there are accelerators in the cloud which are not available to purchase by the general public and for a time the same applied also to GPUs in a practical sense (i.e., GPUs were being sold in principle but were out of stock for months on end). Depending on your individual time value of money, it can make sense to rent even a high rate so you can access this hardware now.
  - ip26 5y ago
    I'm excited on behalf of the scientists I know for cloud HPC. I see a lot of "develop program locally on tiny subset of data" followed by "how the hell am I ever going to run the full model for the final run on all the data!?" A couple hours machine time leased on a behemoth is really the ticket, and not really that expensive.
    - 5y ago

BeeOnRope 5y ago

I wrote this, happy for any feedback or to answer any question about the methodology or otherwise. One thing note is that the headline difference between these instances types is the CPU architecture, the value offered by these instance types is as much about how many IOPS and how much SSD space EC2 chooses to offer for each family, especially for this IO-bound benchmark. It is entirely plausible that EC2 will offer more hardware bang-for-buck on Graviton instances in an effort to encourage adoption so we can't necessarily draw a strong conclusion about the future of the Arm vs x86 battle on the server CPU front since Amazon is not an Arms-length entity here (pun intended).

Uehreka 5y ago
One bit of feedback, it’s a good idea to put “higher is better” or “lower is better” on graph axes.
- BeeOnRope 5y ago
  Thanks, this is great feedback that I'll implement next time. You are right that it is not always obvious especially when talking about both throughput (higher is better) and latency (lower is better) in the same post.
  - bpicolo 5y ago
    I might be missing the context, but I think the units on the graphs might be off? Currently "Bytes per second". Guessing there should be a multiplier on that?
    - BeeOnRope 5y ago
      Indeed, the multiplier is 1e9. It was there in the original matplotlib charts, but was dropped in the "chart beautification" process, probably due to the way mpl likes to hide this detail off in a corner somewhere.

CyanLite2 5y ago

cliff notes? Site seems to have been hugged to death

pzarex 5y ago
Link works for me... TL;DR, Arm instances give better bang for the buck...
speed_spread 5y ago
Probably hosted on Graviton instances

sharpy 5y ago

Recently worked on migrating a number of services (largely CPU bound) to Graviton 2. It was a bit of hit and mess. For some services, it worked great. Very negligible hit to performance, and since equivalent Graviton2 instances cost 20% less, it was a no brainer. Others needed a little bit of rewrite. There were also a couple that just didn't see good performance on ARM (very branchy code)

BeeOnRope 5y ago
Interesting. Other than branchy-ness, was there anything else you think correlated to poorer performance on Graviton 2.
- ceeplusplus 5y ago
  Branchy-ness usually also correlates with cache friendliness (e.g. branchy code is usually a server app calling lots of different functions, with lots of virtual function calls), and Graviton2 has an abnormally small L3 cache relative to its x86 competitors. If I had to hazard a guess I'd say heavily vectorized code will also perform worse especially compared to Intel instances where you have AVX-512.
  - winrid 5y ago
    Didn't read into this much yet but the smaller L3 cache makes me sad. Benchmarking will show true colors but hoping to move some DB clusters to Gravaton2.

redwood 5y ago

Got to feel bad for Intel

phonon 5y ago

You're comparing 4th generation storage to third generation. Unfortunately, the I4i instances are not publically available, despite being announced 4 months ago :-( https://press.aboutamazon.com/news-releases/news-release-details/aws-announces-three-new-amazon-ec2-instances-powered-aws https://press.aboutamazon.com/news-releases/news-release-det... https://aws.amazon.com/ec2/instance-types/i4i/ https://aws.amazon.com/ec2/instance-types/i4i/