7 ms·

AWS unveils Graviton4 & Trainium2

90 points by skilled 3y ago

fhub 3y ago
Not much to discuss until there is pricing. I have a bunch of Graviton2 instances that didn't make sense to upgrade to any Graviton3 instances due to pricing bump for 16GB 4 cores (t4g.xlarge).
monlockandkey 3y ago
What Arm core is Graviton 4 using? 30% performance uplift is a good amount
- cherioo 3y ago
  Likely Neoverse V2 architecture, based on A710 cores
  - monlockandkey 3y ago
    Isn't it based on Cortex X3?
- aeyes 3y ago
  > Neoverse V2 https://aws.amazon.com/blogs/aws/join-the-preview-for-new-memory-optimized-aws-graviton4-powered-amazon-ec2-instances-r8g/ https://aws.amazon.com/blogs/aws/join-the-preview-for-new-me...
PedroBatista 3y ago
What happens to old&used Graviton 3 chips? At least in the "old days" there was ( still is ) a secondary market for used server parts.. Don't know how companies like Amazon, Microsoft and Google would frame a question like this so their "green" narratives wouldn't be hurt but I'm sure they'll do an excellent job.
- rstupek 3y ago
  They'll continue to run in their datacenter since they're still basically brand new?
- baz00 3y ago
  They're still running prehistoric Intel Xeons. I'm sure they'll just rot slowly until the instances fail.
  - jamesfinlayson 3y ago
    This - I've been seeing recent Fargate workloads mysteriously scaling due to high CPU even though there's no traffic. I started logging the CPU as part of task start-up and I've seen five year old Xeons running my workloads.
    - dilyevsky 3y ago
      5 years aint too bad, on google their standard n1 class still runs Haswell in some cases which came out 10+ years ago
      - jamesfinlayson 3y ago
        Ouch. Possibly some are older - there were a few processor names that I couldn't find any release date for.
        dilyevsky 3y ago
        I ran cpu benches there a while ago and it actually performed better wrt single core performance than the newer shit so not all bad
        imglorp 3y ago
        Who cares, if it's priced appropriately? If I just need a generic x86 t2.nano or whatever, why not? They will run it til the card dies, chuck it in the bin when it does, and until then they can pass some of the savings to users hopefully, win win?
buildbot 3y ago
The scale they are quoting at 100,000 chip clusters and 65 exaflops seems impossible. At 800W per chip, that's 80MW of power! Unless they literally built an entire DC of these things, nobody is training anything on the entire cluster at once. It's probably 10-20 separate datacenters being combined for marketing reasons here.
- tempay 3y ago
  What makes you think it's 800W per chip?
  - buildbot 3y ago
    It's about what the I though the H100 was, that's 700W actually. But even at say, 400W, that's 40MW of power. I guess some datacenters are built in the 40-100MW range from some quick googling, but I really doubt they actually can network 100,000 chips together in any sort of performant way, that's supercomputer level interconnect. I don't think most datacenters support highly interlinked network interconnect like this would need either.
    - tempay 3y ago
      They have instances with 16 chips so I presume there are at least 16 chips per server. I'd also expect the power consumption to be more like 100-200W given they seem more like Google's TPUs than a H100. For the interconnect I doubt this is their typical interconnect but it doesn't seem completely unreasonable. Even when not running massive clusters they'll still need the interconnect to pair the random collections of machines that people are using.
      - buildbot 3y ago
        I don’t know - apparently they are watercooling this gen: https://www.servethehome.com/aws-graviton4-is-an-even-bigger-arm-server-processor-and-tranium2-for-ai-nvidia/ https://www.servethehome.com/aws-graviton4-is-an-even-bigger... You don’t watercool 200W chips typically, and you can in theory air cool 8x 800 watt nvidia h100s in a single system. These are also 4-5u systems! 16 chips in one node would be ambitious, I would expect the 16 chip offering to really be several closely located nodes in the same rack/nearby.
LunaSea 3y ago
Since Graviton3 still isn't available in most regions, especially on the RDS side, I'm really not holding my breath.
ilaksh 3y ago
Do you need specific software to train a model using Trainium2? For example, what about fine-tuning a language model? Will something like QLoRA work?
snewman 3y ago
> Graviton4 processors deliver up to 30% better compute performance, 50% more cores, and 75% more memory bandwidth than Graviton3. This seems ambiguous. Presumably this is 50% more cores per chip. What about "30% better compute performance" and "75% more memory bandwidth": is that per core, or per chip? If the latter, then per-core compute performance would actually be lower. Also, "up to" could be hiding almost anything. Has anyone seen a source with clearer information as to how per-core application performance compares to earlier Graviton generations?
- p1esk 3y ago
  Wait, how could "50% more cores, and 75% more memory bandwidth" result in anything less than 50% of better compute performance?
  - jagger27 3y ago
    Clock speed
  - snewman 3y ago
    Because AWS doesn't rent you chips, it rents you cores. If a chip has 50% more cores, that doesn't help you as a renter of cores, just means they get to rent out more cores. Of course it might give Amazon room to offer these instances at a lower hourly rate per core, which would ultimately cash out as improved cost/performance for AWS customers.
- bluedino 3y ago
  It's the best Graviton processor yet!
- kevincox 3y ago
  I would assume that "up to" means that for all of the workloads that they benchmarked the best result was 30% better compute performance. Not a very useful number as your workload is very unlikely to hit the right set of requirements to see that uplift.
- otterley 3y ago
aseipp 3y ago
Neoverse V2, so this will be probably be the first widely available ARMv9 server with SVE2, a server-class SKU you can actually get your hands on (i.e. not a mobile phone/Grace/Fugaku.) It's about damn time!
ksec 3y ago
It is interesting how late the Cortex X3, ( Neoverse V3 or Graviton 4 ) arrives on server when the X4 is already being used and close to shipping in millions within months or weeks. End of Next Year we will get 128 Core Neoverse V4 / Cortex X4 with 3nm. And 3nm Zen 5 EPYC. Server CPU Market is getting quite exciting.
snird 3y ago
One of the employees in the team developing Graviton4 is kidnapped by Hamas in Gaza: https://www-geektime-co-il.translate.goog/amazon-aws-employees-for-sasha-trufanov/?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp https://www-geektime-co-il.translate.goog/amazon-aws-employe...
- insanitybit 3y ago
  > This web property is not accessible via this address.
ThinkBeat 3y ago
It feels a bit weird with MS, AWS probably other cloud owners develop their own CPUS and AI oriented chips and telling the world about the specs. Yet nobody will ever get noe to play with them in RL. I cant hope to buy one a year from now and stuff it in my home office. What will all this mean for consumer oriented cpus? Would it be accurate to say that Intel funds part of the development of consumer cpus with the server cpus? (or is it the other way around). It seems like Xeon chip advances drip downwards after a while. If AWS and Azure stop buying chips from Intel and AMD, presumably that woud be interesting.
resters 3y ago
In my opinion the key takeaway is that compute is becoming commoditized much more rapidly than anyone expected, and that IP is becoming less and less relevant compared to fabrication, energy and land. For consumers of cloud infrastructure, there is little concern how many teraflops per cubic meter or per watt hour other than very locality-specific edge use-cases. Laptops and phones are are already a SoC with IO in a particular form factor, and sever farms will go in the same direction with minor differences in the energy or rack density that come out in the wash.
jauntywundrkind 3y ago
Any guesses what the various chips on the package are? I'd guess maybe the two directly abutting the core are memory controllers, but maybe they are the stacked memory? Maybe the top and bottom chips are io controllers? It felt like destiny that eventually Nitro was going to be on-package, maybe those are basically big honking nitro-like chips?

90 points by skilled 3y ago

fhub 3y ago

Not much to discuss until there is pricing. I have a bunch of Graviton2 instances that didn't make sense to upgrade to any Graviton3 instances due to pricing bump for 16GB 4 cores (t4g.xlarge).

monlockandkey 3y ago

What Arm core is Graviton 4 using? 30% performance uplift is a good amount

cherioo 3y ago
Likely Neoverse V2 architecture, based on A710 cores
- monlockandkey 3y ago
  Isn't it based on Cortex X3?
aeyes 3y ago
> Neoverse V2 https://aws.amazon.com/blogs/aws/join-the-preview-for-new-memory-optimized-aws-graviton4-powered-amazon-ec2-instances-r8g/ https://aws.amazon.com/blogs/aws/join-the-preview-for-new-me...

PedroBatista 3y ago

What happens to old&used Graviton 3 chips? At least in the "old days" there was ( still is ) a secondary market for used server parts.. Don't know how companies like Amazon, Microsoft and Google would frame a question like this so their "green" narratives wouldn't be hurt but I'm sure they'll do an excellent job.

rstupek 3y ago
They'll continue to run in their datacenter since they're still basically brand new?
baz00 3y ago
They're still running prehistoric Intel Xeons. I'm sure they'll just rot slowly until the instances fail.
- jamesfinlayson 3y ago
  This - I've been seeing recent Fargate workloads mysteriously scaling due to high CPU even though there's no traffic. I started logging the CPU as part of task start-up and I've seen five year old Xeons running my workloads.
  - dilyevsky 3y ago
    5 years aint too bad, on google their standard n1 class still runs Haswell in some cases which came out 10+ years ago
    - jamesfinlayson 3y ago
      Ouch. Possibly some are older - there were a few processor names that I couldn't find any release date for.
      - dilyevsky 3y ago
        I ran cpu benches there a while ago and it actually performed better wrt single core performance than the newer shit so not all bad
      - imglorp 3y ago
        Who cares, if it's priced appropriately? If I just need a generic x86 t2.nano or whatever, why not? They will run it til the card dies, chuck it in the bin when it does, and until then they can pass some of the savings to users hopefully, win win?

buildbot 3y ago

The scale they are quoting at 100,000 chip clusters and 65 exaflops seems impossible. At 800W per chip, that's 80MW of power! Unless they literally built an entire DC of these things, nobody is training anything on the entire cluster at once. It's probably 10-20 separate datacenters being combined for marketing reasons here.

tempay 3y ago
What makes you think it's 800W per chip?
- buildbot 3y ago
  It's about what the I though the H100 was, that's 700W actually. But even at say, 400W, that's 40MW of power. I guess some datacenters are built in the 40-100MW range from some quick googling, but I really doubt they actually can network 100,000 chips together in any sort of performant way, that's supercomputer level interconnect. I don't think most datacenters support highly interlinked network interconnect like this would need either.
  - tempay 3y ago
    They have instances with 16 chips so I presume there are at least 16 chips per server. I'd also expect the power consumption to be more like 100-200W given they seem more like Google's TPUs than a H100. For the interconnect I doubt this is their typical interconnect but it doesn't seem completely unreasonable. Even when not running massive clusters they'll still need the interconnect to pair the random collections of machines that people are using.
    - buildbot 3y ago
      I don’t know - apparently they are watercooling this gen: https://www.servethehome.com/aws-graviton4-is-an-even-bigger-arm-server-processor-and-tranium2-for-ai-nvidia/ https://www.servethehome.com/aws-graviton4-is-an-even-bigger... You don’t watercool 200W chips typically, and you can in theory air cool 8x 800 watt nvidia h100s in a single system. These are also 4-5u systems! 16 chips in one node would be ambitious, I would expect the 16 chip offering to really be several closely located nodes in the same rack/nearby.

LunaSea 3y ago

Since Graviton3 still isn't available in most regions, especially on the RDS side, I'm really not holding my breath.

ilaksh 3y ago

Do you need specific software to train a model using Trainium2? For example, what about fine-tuning a language model? Will something like QLoRA work?

snewman 3y ago

> Graviton4 processors deliver up to 30% better compute performance, 50% more cores, and 75% more memory bandwidth than Graviton3. This seems ambiguous. Presumably this is 50% more cores per chip. What about "30% better compute performance" and "75% more memory bandwidth": is that per core, or per chip? If the latter, then per-core compute performance would actually be lower. Also, "up to" could be hiding almost anything. Has anyone seen a source with clearer information as to how per-core application performance compares to earlier Graviton generations?

p1esk 3y ago
Wait, how could "50% more cores, and 75% more memory bandwidth" result in anything less than 50% of better compute performance?
- jagger27 3y ago
  Clock speed
- snewman 3y ago
  Because AWS doesn't rent you chips, it rents you cores. If a chip has 50% more cores, that doesn't help you as a renter of cores, just means they get to rent out more cores. Of course it might give Amazon room to offer these instances at a lower hourly rate per core, which would ultimately cash out as improved cost/performance for AWS customers.
bluedino 3y ago
It's the best Graviton processor yet!
kevincox 3y ago
I would assume that "up to" means that for all of the workloads that they benchmarked the best result was 30% better compute performance. Not a very useful number as your workload is very unlikely to hit the right set of requirements to see that uplift.
otterley 3y ago

aseipp 3y ago

Neoverse V2, so this will be probably be the first widely available ARMv9 server with SVE2, a server-class SKU you can actually get your hands on (i.e. not a mobile phone/Grace/Fugaku.) It's about damn time!

ksec 3y ago

It is interesting how late the Cortex X3, ( Neoverse V3 or Graviton 4 ) arrives on server when the X4 is already being used and close to shipping in millions within months or weeks. End of Next Year we will get 128 Core Neoverse V4 / Cortex X4 with 3nm. And 3nm Zen 5 EPYC. Server CPU Market is getting quite exciting.

snird 3y ago

One of the employees in the team developing Graviton4 is kidnapped by Hamas in Gaza: https://www-geektime-co-il.translate.goog/amazon-aws-employees-for-sasha-trufanov/?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp https://www-geektime-co-il.translate.goog/amazon-aws-employe...

insanitybit 3y ago
> This web property is not accessible via this address.

ThinkBeat 3y ago

It feels a bit weird with MS, AWS probably other cloud owners develop their own CPUS and AI oriented chips and telling the world about the specs. Yet nobody will ever get noe to play with them in RL. I cant hope to buy one a year from now and stuff it in my home office. What will all this mean for consumer oriented cpus? Would it be accurate to say that Intel funds part of the development of consumer cpus with the server cpus? (or is it the other way around). It seems like Xeon chip advances drip downwards after a while. If AWS and Azure stop buying chips from Intel and AMD, presumably that woud be interesting.

resters 3y ago

In my opinion the key takeaway is that compute is becoming commoditized much more rapidly than anyone expected, and that IP is becoming less and less relevant compared to fabrication, energy and land. For consumers of cloud infrastructure, there is little concern how many teraflops per cubic meter or per watt hour other than very locality-specific edge use-cases. Laptops and phones are are already a SoC with IO in a particular form factor, and sever farms will go in the same direction with minor differences in the energy or rack density that come out in the wash.

jauntywundrkind 3y ago

Any guesses what the various chips on the package are? I'd guess maybe the two directly abutting the core are memory controllers, but maybe they are the stacked memory? Maybe the top and bottom chips are io controllers? It felt like destiny that eventually Nitro was going to be on-package, maybe those are basically big honking nitro-like chips?